Search: “spatial-video”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control

PhysStream autoregressive video model enables physics-grounded mid-generation motion control, cutting trajectory error 12% and FVMD 33% versus strongest baselines.

PhysStream is an autoregressive image-to-video model that incorporates structured scene memory—positional maps and object tracking maps derived online from previously generated frames—and supports fine-grained motion control via sparse velocity-increment signals encoding physical quantities. Training proceeds in two stages: a bidirectional model finetuned with motion-control conditioning, then a causal autoregressive model with scene memory. It reduces motion distribution distance (FVMD) by 33% and trajectory error by 12% over the strongest baselines, and human evaluators prefer it in over 85% of in-the-wild comparisons.

arXiv cs.AI / cs.LG / cs.CL·Sep 15, 17:55 UTC · 1d agoAI research 2 sources

25

SpatialBlock: Enhancing Spatial Intelligence in LVLMs via Synthetic Block-Stacking Problem

Researchers release SpatialBlock-15k, a synthetic block-stacking dataset that improves 3D spatial reasoning in large vision-language models without dense geometric annotations.

The paper addresses limited spatial intelligence in LVLMs by training on structured block-manipulation tasks instead of costly real-scene annotated datasets. SpatialBlock-15k contains 15,000 synthetic problems covering 3D-to-2D projection, viewpoint transformation, and structural combination, with color modulation as visual cues. LVLMs trained on it via direct answering or reasoning-based prediction outperform baselines and generalize to real-world spatial tasks. Code and data are released on GitHub.

Hugging Face daily papers·Sep 6, 20:00 UTC · 9d agoAI research

28

AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video

AlayaVista is a camera-controllable streaming video world model that decouples panoramic scene evolution from perspective synthesis, trained on a 1,318-hour 4K dataset.

AlayaVista builds a 360-degree scene prior from a single perspective image, evolves it as a camera-conditioned panoramic latent state, and maps it to perspective video via a latent viewport renderer plus a perspective refiner. Chunk-autoregressive generation and few-step distillation enable efficient streaming. The authors introduce MUGEN, a real-world panoramic video dataset with 1,318 hours of at-least-4K video and rich semantic and geometric annotations.

Hugging Face daily papers·Sep 12, 20:00 UTC · 3d agoAI research

45

Multi-Grid Post-Training for Long-Form Multi-Shot Video Generation

MovieGrid arranges long videos on spatial grids during post-training, generating 6.05x more shots than temporal packing with state-of-the-art cross-shot consistency.

MovieGrid is a multi-grid post-training paradigm that decomposes long videos into temporally ordered chunks arranged on a spatial grid for joint modeling, enabling cross-chunk information exchange. The authors build the Multi-Grid Long Video (MGLV) dataset from 1,000 long-form videos, producing 54K grid videos paired with character-aware story prompts. Under the same token budget, MovieGrid generates 6.05x more shots than Temporal Packing in a 1,616-frame video. It achieves state-of-the-art intra-shot consistency of 0.9131 versus 0.8086 for HoloCine and inter-shot consistency of 0.5914 versus 0.5384 for StoryMem.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

20

World in World: Explore the World with World Models

World in World is a training-free inference-time interface enabling camera-controlled rerendering, long-horizon revisiting, and motion transfer with frozen causal video world models.

The framework converts heterogeneous control evidence into camera- and time-labelled clean visual states that are read through the native self-attention of a frozen causal video model. Evidence includes source-video observations, target-view scene projections, geometry renderings for completing newly exposed regions, and retrieved generated states beyond the rolling cache. A correspondence router combines persistent point identities with geometry for token correspondences, while evidence-wise attention CFG independently regulates each auxiliary channel. The shared interface supports camera-controlled rerendering, long-horizon revisiting, and human-motion transfer without additional training.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

30

Diffusion TV: Experiencing Diffusion Models through Tangible, Embodied Interaction

Researchers present Diffusion TV, a CRT-based installation where antenna manipulation lets audiences physically experience diffusion model denoising.

Diffusion TV is an interactive installation built around a modified CRT television where turning the antenna controls the clarity of AI-generated images and sounds, mirroring the denoising process of diffusion models. Three channels present AI-generated animals from the past, present, and future within a temporal and ecological narrative. The authors frame the work as an embodied, non-verbal alternative to explainable AI that highlights intermediate generative states rather than final outputs.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:51 UTC · 12d agoAI research

12

BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender

Blender-VideoBench evaluates agentic video understanding by having agents programmatically reconstruct real videos in Blender scenes.

BVB (Blender-VideoBench) tests whether multimodal agents truly understand videos by requiring programmatic reconstruction of real-world videos as animated Blender scenes via a lightweight Mini-BVB harness under identical sandbox and cost constraints. Evaluation uses Dual VQA for spatiotemporal fact preservation and Latent Similarity for perceptual match, combined in a square-root mean overall score. Across 51 configurations from 10 model families, the best model reaches 88.6 Latent Similarity but retains only 53.7% of source-correct spatiotemporal answers, showing semantic retention remains the main challenge.

Hugging Face daily papers·Sep 13, 20:00 UTC · 2d agoAI research

30

A Chosen Future Can Still Be Rewritten: Causal Writability in Video Models

Study shows video models often learn correct physics but fail to use it; low-dimensional 'causal writability' edits can restore correct motion.

The paper demonstrates 'causal writability' in video generation models: physically correct motion remains available inside the model even when the model outputs incorrect motion. In a red/blue mass oscillation setup, a low-dimensional edit predicted from simple physical variables restores correct fast motion, with a sharp depth boundary marking commitment. Early causal writability predicts which training errors later get corrected, and both writability and closure reproduce in a pretrained 1.3B video model.

arXiv cs.AI / cs.LG / cs.CL·Sep 14, 17:58 UTC · 2d agoAI research

33

Unfold The World: Factorize 4D Properties in Reinforcing Spatial Reasoning

FactoSR factorizes 4D spatial reasoning into XY, Z, and T reinforcement-learning sub-objectives, boosting VLM performance on VSI-Bench by 5.9% and All-Angles-Bench by 4.5%.

Researchers present FactoSR, a factorized reinforcement learning framework that decomposes world-consistent reasoning into planar correspondence, depth consistency, and temporal reversibility sub-objectives. Optimizing these verifiable constraints turns the ill-posed projection recovery problem into tangible reasoning steps. Evaluations show gains of 5.9% on VSI-Bench and 4.5% on All-Angles-Bench for 3D and 4D reasoning, arguing VLMs' spatial bottleneck stems from training on 2D projections versus latent 3D geometry and temporal continuity.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

26

SynthGait-19K: A Physically Grounded Synthetic Video Dataset for Gait Parameter Estimation

Researchers release SynthGait-19K, a synthetic video dataset with 19,272 walking videos for training gait parameter estimation models.

SynthGait-19K is a physically grounded synthetic video dataset built from 6,427 MoCap sequences of 437 subjects, yielding 19,272 walking videos with SMPL motion and annotations for six gait parameters. The authors introduce Gait2Vid, a pipeline that unifies heterogeneous MoCap recordings and synthesizes RGB videos under controllable viewpoints, validating gait events against force-platform measurements. Using the dataset they benchmark direct RGB, pose-based, biomechanical, and human-mesh-recovery approaches, and introduce GaitXFormer as a direct RGB reference model. Findings show synthetic supervision transfers to real video, while spatial gait parameters are more sensitive to visual domain shift.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

28

Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout

Researchers propose Mask Forcing, a dual-noise masking rollout that mitigates mode collapse in autoregressive video diffusion distillation.

The paper targets over-saturation and over-smoothing in autoregressive video diffusion models distilled via Distribution Matching Distillation, attributed to reverse-KL mode-seeking behavior. Mask Forcing perturbs the student self-rollout with random masks along spatial and temporal axes, injecting cleaner tokens that act as denoising guidance for noisier tokens. Experiments show improved visual quality across multiple distillation methods without using real video data or extra post-training stages.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

22

TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Model

Researchers introduce TANGO, a whole-body vision-language-action model enabling zero-shot language-guided humanoid navigation on the Unitree G1 robot.

TANGO addresses humanoid navigation in cluttered indoor environments by predicting 29-DoF joint-space actions directly from natural-language instructions and egocentric RGB, rather than 2D path planning. It is trained entirely in simulation via a pipeline combining global path planning, kinematic whole-body motion generation, obstacle-aware motion editing, and RL-based tracking. In simulation it achieves state-of-the-art vision-language navigation performance and transfers zero-shot to a Unitree G1 humanoid without any real-world navigation data.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research1

27

FreeFlow: A Bias-free Hierarchical Transformer for Optical Flow Estimation

FreeFlow is a bias-free hierarchical transformer achieving state-of-the-art optical flow results on Sintel, KITTI-2015, and Spring benchmarks.

FreeFlow replaces task-specific inductive biases like correlation volumes and iterative warping with a single feed-forward encoder-decoder combining window, shifted-window, and reduced-resolution global attention. It reaches 0.68/1.48 EPE on Sintel Clean/Final, 3.23 Fl-all on KITTI-2015, and 3.192 1px on Spring. The architecture scales consistently from small to large variants and remains memory efficient at 1080p inference.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

30

CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements

Researchers release CosmoH2G, a 6,189-episode hand-to-gripper dataset with a two-stage method for complex spatial robot manipulation.

The paper introduces a scalable acquisition pipeline using a handheld gripper to collect paired hand-gripper demonstrations, producing 6,189 episodes across 1,254 unique objects with higher spatial complexity than existing benchmarks. A two-stage framework first predicts sparse gripper keyframes (initial and terminal), then generates the full continuous action sequence conditioned on them, while learning gripper orientation and post-optimizing translation via grasping heuristics and kinematic consistency. Simulation and real-robot experiments show stable, precise hand-to-gripper transfer of complex spatial manipulations, outperforming traditional baselines.

Hugging Face daily papers·Sep 6, 20:00 UTC · 9d agoAI research

24

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

A survey catalogs inference-efficiency techniques for video and audiovisual LLMs, mapping bottlenecks in sampling, encoding, token reduction, and LLM decoding.

This survey covers inference-efficiency mechanisms for visual and audiovisual video LLMs, reporting reductions in parameters, FLOPs, latency, memory, and token counts. It organizes methods by pipeline stage, covering frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding for systems built since late 2022. The authors compile accuracy-cost comparisons under shared host models and input protocols, identify gaps in audiovisual efficiency and standardized evaluation, and maintain a public repository.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

40

TANGO: Humanoid Navigation in Cluttered Environments with a Whole-Body Vision-Language-Action Model

Researchers present TANGO, a whole-body vision-language-action model enabling humanoid robots to traverse cluttered spaces from language instructions.

TANGO predicts 29-DoF joint-space actions from egocentric RGB observations and natural-language instructions for whole-body humanoid navigation, going beyond 2D path planning. It is trained entirely in simulation using global path planning, kinematic whole-body motion generation, obstacle-aware motion editing, and RL-based tracking. The model reports state-of-the-art simulation performance and was deployed zero-shot on a Unitree G1 humanoid without any real-world navigation training data.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:59 UTC · 8d agoAI research1

28

3D Point Splatting for mmWave Radar Novel View Synthesis

Researchers propose 3DPS, a differentiable point renderer for mmWave radar novel view synthesis that outperforms optical-NVS baselines by 1.7x-5.2x.

The paper introduces 3D Point Splatting (3DPS), the first differentiable point renderer for radar, derived from the solid-angle form of the radar equation with ITU-R P.2040 material models and complex phasor splatting. On six outdoor ColoRadar scenes it reaches 0.587 mean Pearson correlation on held-out range-azimuth images, between 1.7x and 5.2x the RadarSplat, Radar Fields, and DART baselines. The same optimized scene produces ADC, complex range profile, and RA outputs via standard FFT pipelines, and training takes about 3 minutes per scene on an RTX 4090.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:52 UTC · 6d agoAI research

28

TransNormal-2: Geometry-Grounded Rectified Flow with Edge-Aware Decoding for Precise Normal Estimation

TransNormal-2 improves monocular surface-normal estimation by fixing VAE edge degradation with geometry-aware losses and refinement, matching MoGe-2 with 1.4% of annotations.

TransNormal-2 is a FLUX.2-based rectified-flow framework for monocular surface-normal estimation with single-step deterministic inference. The authors quantify that VAE 8x spatial compression introduces 1.3-8.5 degrees of mean angular error even on ground-truth normals, with edge error up to 2.8x the global error. The method adds geometry-aware pixel-space losses and an RGB-guided Geometric Refinement Module to correct boundary-localized decoding errors. It matches or exceeds MoGe-2 on all eight reported metrics using only 1.4% as many task-specific annotations, and cuts transparent-object MAE by 4.2 degrees on ClearGrasp and 3.1 degrees on ClearPose.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

20

Attention-DP3: Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioning

Attention-DP3 adds spatially object-aware attentional conditioning to 3D diffusion policies, improving robotic manipulation by up to 31% under heavy clutter.

Attention-DP3 injects object-level geometric cues into the unchanged DP3 diffusion policy via Tri-field Attentional Conditioning, using targetness, intra-target saliency, and backgroundness fields. Open-vocabulary 2D segmentation masks are lifted to 3D with calibrated camera geometry to build object-centric priors. Experiments on Adroit, DexArt, MetaWorld, and a real-world SO101 platform show state-of-the-art results, outperforming DP3 by up to 31% under heavy distractor clutter; the code is publicly available on GitHub.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

15

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled pure-autoregressive testbed shows task-specific validation losses rank image tokenizers differently, with I2T loss the most consistent signal.

Researchers built a controlled pure-autoregressive testbed and tracked task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, correlates consistently with both generation and visual understanding performance after supervised finetuning. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:57 UTC · 8d agoAI research1

30

The Attention Triangle in Audio-Video Models

Researchers analyze the 'attention triangle' in audio-video diffusion models, showing bias-driven cross-attention routing causes semantic leakage and proposing inference-time interventions that improve grounding.

A study probes the three cross-attention edges linking text, audio, and video streams in audio-video diffusion models. It finds the audio-video edge is bidirectional and shaped by parameter-encoded biases, so prompts in tension with learned priors can be overridden, producing visually canonical but incorrect outputs. Attention-derived signals are used as diagnostics and to guide inference-time interventions that improve cross-modal semantic grounding while preserving generation quality.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

24

SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

SceneMosaic combines image-based 3D priors with VLM agent refinement to generate diverse, simulation-ready indoor scenes 24x faster than agentic baselines.

SceneMosaic is a hybrid framework that takes an initial candidate from a learned image-to-3D prior and evolves it with VLM agents for efficiency and physical validity. It decomposes scenes into independent local units, evolves each separately, and composes the global scene via Cartesian product. On SceneEval-100 it matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Code is publicly available.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research

25

Vidu S2: Real-Time Interactive, Editable, and Spatial Video Generation

Vidu S2 delivers real-time 720p interactive avatar generation plus real-time video editing with style, clothing, character, and background replacement.

Vidu S2 comprises Vidu S2-Avatar, a real-time interactive digital-character model, and Vidu S2-Editing, a real-time video editing model. S2-Avatar supports real-time 720p generation, dynamic references updatable at any moment, and stronger instruction following such as dancing, while S2-Editing performs real-time style rendering, clothing replacement, character replacement, and background replacement. The work also explores real-time spatial video generation for both models, reports outperforming all baselines, and offers a playable online demo at vidu.com.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoModel release

40

Recursive Code World Models: Building Complex Worlds through Recursive Scene Programs

RCWM reconstructs complex 3D worlds as executable code from a single image using recursive scene programs with global-local-global solver recursion.

The paper introduces Recursive Code World Models, coupling a Recursive Scene Program representation with a recursive construction solver for image-to-3D-world reconstruction. Each solver call establishes the whole scene, recursively reconstructs unresolved parts, and revisits the whole to refine composition, while a vision-language coding agent compares reference images with scene renders to guide refinement. RCWM outperforms prior code-based image-to-scene reconstruction methods, and ablations show deeper recursive calls improve fine-scale reconstruction.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research1

33

Agentic Visual Generation: From Generative Models to Agentic Control

Researchers propose an L0-L4 control taxonomy for agentic visual generation, classifying controllers from fixed conditioning to experience-adaptive decision-making.

This paper proposes a taxonomy for agentic visual generation organized by what the controller can directly control in the generation process, rather than by planning depth, tool count, or model size. Levels range from L1 Conditioning Control through L2 Execution Control, L3 Outcome-Adaptive Control, and L4 Experience-Adaptive Control, with L0 Fixed Support denoting systems without a deployed decision-making controller. The framework is applied across image, video, editing, 3D, world, slide, and user-interface generation to map how controller capabilities and mechanisms have evolved across the field.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

22

OracleZoom: On-Policy Self-Distillation Inspired Reference-Constrained Recursive Image Super Resolution

OracleZoom enables recursive extreme-scale image super-resolution via reference-constrained on-policy distillation, reducing hallucinations at deep zoom scales.

OracleZoom tackles recursive super-resolution, where repeated feeding of predictions back into the same model leaves deeper-scale outputs unsupervised as required source resolution grows geometrically. The framework trains on its own trajectory while carrying the last ground-truth evidence beyond the supervision boundary, combining direct and cross-scale supervision, a no-reference quality objective, a KL-constrained pretrained latent prior, and EMA consistency. Across seven datasets it achieves state-of-the-art SR quality across zoom scales, averaging 0.713 CLIPIQA with larger gains at deeper scales and significantly reduced hallucinations. Code, data, and models are publicly released.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

20

RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting

RelightFormer is a feed-forward generative transformer for photorealistic single- and multi-view object relighting, trained on a 90K-object dataset.

Researchers introduce RelightFormer, a feed-forward generative transformer adapted from a video foundation model that performs direct image relighting without explicit intrinsic property estimation. The architecture injects target environment maps via a latent illumination module with cross-attention and uses permutation-invariant positional encodings for unordered multi-view inputs. Training relies on the newly constructed Laval Objaverse Dataset (LOD) with 90K objects and 39K unique illuminations, and the model shows state-of-the-art quality with strong zero-shot generalization across single-view, multi-view, and novel-view relighting.

Hugging Face daily papers·Sep 6, 20:00 UTC · 9d agoAI research

38

LynnReal-Omni: Native multi-modal Video Generation for Agentic Visual Workflows

LynnReal-Omni unifies controllable video generation tasks in a 32B multimodal diffusion transformer, with a 27B Flash variant rendering 540p clips in 377 ms.

LynnReal-Omni is a native multimodal video generation framework built on a 32B shared multimodal diffusion transformer unifying text-to-video, image-conditioned generation, reference guidance, structural control, editing, restoration and long-video generation, accepting heterogeneous inputs like 3D renders and game recordings for agentic visual workflows. A dedicated 27B Flash model enables real-time rendering, producing a 22-frame 540p video in 377 ms on one H100 versus 843 ms for the full model. The work introduces a curated multi-shot audiovisual data pipeline and MSAVP, a 100-prompt, 20-metric evaluation design covering instruction following, plausibility, visual quality, temporal behavior and audio coordination.

Hugging Face daily papers·Sep 13, 20:00 UTC · 2d agoAI research

28

ActionSplice: In-Flight Action Editing for Interactive World Models

ActionSplice enables in-flight action editing in chunk-autoregressive video world models via a lightweight corrector, avoiding rollback or waiting for the next chunk.

ActionSplice is an inference framework that formulates in-flight action editing for chunk-autoregressive video world models as Counterfactual State Transport (CST), where a lightweight corrector transports the interrupted backbone-native representation toward the matched state induced by the revised action. The world model and sampler remain frozen, and sampling resumes without replaying completed evaluations. Across minWM-Wan Action2V and HY-WM1.5, the retargeting variant CST-R reduces rollback-relative LPIPS by 61.5% and 75.9% versus direct condition swapping, while the temporal-splicing variant CST-T reduces suffix LPIPS by 56.1% and 77.5% with 2.73x and 1.69x pixel-ready speedups over waiting.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

22