Search: “scene-programs”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control

PhysStream enables mid-generation interactive control of physics-grounded video via structured scene memory and velocity-increment signals, reducing motion distribution distance 33%.

PhysStream is an autoregressive physics-grounded image-to-video model that maintains structured scene memory—positional maps and object tracking maps derived online from previously generated frames—and accepts fine-grained motion control via sparse velocity-increment signals encoding physical quantities. Training runs in two stages: a bidirectional model finetuned with motion-control conditioning, then a causal autoregressive model with structured scene memory. It supports interactive mid-generation control over multi-object tabletop rigid-body scenes, reducing motion distribution distance (FVMD) by 33% and trajectory error by 12% over the strongest baselines. Human evaluators preferred it in over 85% of in-the-wild comparisons.

Hugging Face daily papers·updated Sep 15, 17:55 UTC · 1d ago·first Sep 14, 20:00 UTC · 1d agoAI research 2 sources

28

Recursive Code World Models: Building Complex Worlds through Recursive Scene Programs

RCWM reconstructs complex 3D worlds as executable code from a single image using recursive scene programs with global-local-global solver recursion.

The paper introduces Recursive Code World Models, coupling a Recursive Scene Program representation with a recursive construction solver for image-to-3D-world reconstruction. Each solver call establishes the whole scene, recursively reconstructs unresolved parts, and revisits the whole to refine composition, while a vision-language coding agent compares reference images with scene renders to guide refinement. RCWM outperforms prior code-based image-to-scene reconstruction methods, and ablations show deeper recursive calls improve fine-scale reconstruction.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research1

33

SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

SceneMosaic combines image-based 3D priors with VLM agent refinement to generate diverse, simulation-ready indoor scenes 24x faster than agentic baselines.

SceneMosaic is a hybrid framework that takes an initial candidate from a learned image-to-3D prior and evolves it with VLM agents for efficiency and physical validity. It decomposes scenes into independent local units, evolves each separately, and composes the global scene via Cartesian product. On SceneEval-100 it matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Code is publicly available.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research

25

BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender

Blender-VideoBench evaluates agentic video understanding by having agents programmatically reconstruct real videos in Blender scenes.

BVB (Blender-VideoBench) tests whether multimodal agents truly understand videos by requiring programmatic reconstruction of real-world videos as animated Blender scenes via a lightweight Mini-BVB harness under identical sandbox and cost constraints. Evaluation uses Dual VQA for spatiotemporal fact preservation and Latent Similarity for perceptual match, combined in a square-root mean overall score. Across 51 configurations from 10 model families, the best model reaches 88.6 Latent Similarity but retains only 53.7% of source-correct spatiotemporal answers, showing semantic retention remains the main challenge.

Hugging Face daily papers·Sep 13, 20:00 UTC · 2d agoAI research

30

Multi-Grid Post-Training for Long-Form Multi-Shot Video Generation

MovieGrid arranges long videos on spatial grids during post-training, generating 6.05x more shots than temporal packing with state-of-the-art cross-shot consistency.

MovieGrid is a multi-grid post-training paradigm that decomposes long videos into temporally ordered chunks arranged on a spatial grid for joint modeling, enabling cross-chunk information exchange. The authors build the Multi-Grid Long Video (MGLV) dataset from 1,000 long-form videos, producing 54K grid videos paired with character-aware story prompts. Under the same token budget, MovieGrid generates 6.05x more shots than Temporal Packing in a 1,616-frame video. It achieves state-of-the-art intra-shot consistency of 0.9131 versus 0.8086 for HoloCine and inter-shot consistency of 0.5914 versus 0.5384 for StoryMem.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

20

GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting

GoDeep achieves annotation-free open-vocabulary 3D segmentation by grounding structured image descriptions in language-only embeddings, outperforming CLIP-based lifting on out-of-vocabulary objects.

GoDeep uses a vision-language model purely as a translator, producing structured entity-level image descriptions that are grounded, projected, and aggregated in a general-purpose language-only embedding space, with no 3D training corpus or dedicated 3D encoder required. On ScanNet++ the pipeline is competitive with strong annotation-free baselines, and on a cultural-heritage benchmark a systematic vocabulary correction reverses initial CLIP-based rankings. Language-space embeddings separate genuinely out-of-vocabulary objects more sharply, localize them within scenes, and keep all predictions explainable as discrete text.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:31 UTC · 8d agoAI research

26

VeriScene: Reconstructing Crime Scenes from Legal Evidence via World-Model Agent

Researchers present VeriScene, a world-model agent that reconstructs crime scenes from forensic photos and witness statements with traceable, physically plausible output.

VeriScene orchestrates a world model to fuse forensic photographs and witness statements of varying reliability into cited narratives and physically plausible re-enactment videos. On a 25-scenario benchmark with planted unreliable testimony, it reaches 0.9014 evidence coverage and 0.7217 factual consistency on 20 test scenes. It outperforms an end-to-end multimodal-LLM baseline by 20.35% in factual consistency and 34.88% in temporal coherence at USD 1.82 per scene.

arXiv cs.CR·Sep 8, 07:14 UTC · 8d agoAI research

26

AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video

AlayaVista is a camera-controllable streaming video world model that decouples panoramic scene evolution from perspective synthesis, trained on a 1,318-hour 4K dataset.

AlayaVista builds a 360-degree scene prior from a single perspective image, evolves it as a camera-conditioned panoramic latent state, and maps it to perspective video via a latent viewport renderer plus a perspective refiner. Chunk-autoregressive generation and few-step distillation enable efficient streaming. The authors introduce MUGEN, a real-world panoramic video dataset with 1,318 hours of at-least-4K video and rich semantic and geometric annotations.

Hugging Face daily papers·Sep 12, 20:00 UTC · 3d agoAI research

45

MOONWALK: Mediating Operations with Intent-Evidence-Action Alignment Across Junior-Supervisor Review Workflows in Animation/VFX Pre-Production

MOONWALK introduces an intent-evidence-action alignment workflow for animation/VFX reviews where AI handles administrative coordination while artists keep creative authority.

MOONWALK is a pre-production review system that articulates creative intent into a shared project record, anchors review judgments to grounded evidence, and converts authorized decisions into concrete revision tasks. AI components handle administrative coordination such as flagging missing context and organizing notes, not creative direction. An in-studio study against a chat-only AI interface showed stronger intent alignment, decision traceability, and checklist executability.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:10 UTC · 7d agoAI research

15

Feature Recovery for Object Understanding After Irreversible Fire Damage

TRACE benchmark with 21.4K scenes studies post-fire object understanding; a Feature Recovery Module improves degraded-image retrieval by 12.5% and material recovery by 20.1%.

The paper introduces TRACE, a transformation-aware benchmark with 21.4K real-image-grounded synthetic scenes, 499 object identities across 189 categories, and five tasks covering degraded-object detection, pristine-state recovery, material recovery, description generation, and functional reasoning. Existing models degrade sharply: RF-DETR mAP falls 71% relative from least to most severe level, and InternVL3.5 retrieval R@1 drops from 93.85 to 28.11. The proposed Feature Recovery Module maps degraded encoder features to pristine-aligned representations while keeping the host model frozen, averaging relative gains of 12.5% for retrieval and 20.1% for material recovery across VLM hosts and severity levels.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

30

Training a coding model to paint watercolours with TRL and OpenEnv

Hugging Face tutorial trains a coding model with TRL and OpenEnv to paint watercolours through generated code.

A Hugging Face blog walkthrough uses the TRL reinforcement learning library and the OpenEnv environment framework to train a coding model. The target task is generating code that produces watercolour-style drawings, serving as a hands-on reinforcement learning training example. No article body was available in the feed, so specifics are limited to the title.

Hugging Face Blog·Sep 3, 00:00 UTC · 13d agoAI tools & infra1

18

CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements

Researchers release CosmoH2G, a 6,189-episode hand-to-gripper dataset with a two-stage method for complex spatial robot manipulation.

The paper introduces a scalable acquisition pipeline using a handheld gripper to collect paired hand-gripper demonstrations, producing 6,189 episodes across 1,254 unique objects with higher spatial complexity than existing benchmarks. A two-stage framework first predicts sparse gripper keyframes (initial and terminal), then generates the full continuous action sequence conditioned on them, while learning gripper orientation and post-optimizing translation via grasping heuristics and kinematic consistency. Simulation and real-robot experiments show stable, precise hand-to-gripper transfer of complex spatial manipulations, outperforming traditional baselines.

Hugging Face daily papers·Sep 6, 20:00 UTC · 9d agoAI research

24

Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs

A survey catalogs inference-efficiency techniques for video and audiovisual LLMs, mapping bottlenecks in sampling, encoding, token reduction, and LLM decoding.

This survey covers inference-efficiency mechanisms for visual and audiovisual video LLMs, reporting reductions in parameters, FLOPs, latency, memory, and token counts. It organizes methods by pipeline stage, covering frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding for systems built since late 2022. The authors compile accuracy-cost comparisons under shared host models and input protocols, identify gaps in audiovisual efficiency and standardized evaluation, and maintain a public repository.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

40

ActionSplice: In-Flight Action Editing for Interactive World Models

ActionSplice enables in-flight action editing in chunk-autoregressive video world models via a lightweight corrector, avoiding rollback or waiting for the next chunk.

ActionSplice is an inference framework that formulates in-flight action editing for chunk-autoregressive video world models as Counterfactual State Transport (CST), where a lightweight corrector transports the interrupted backbone-native representation toward the matched state induced by the revised action. The world model and sampler remain frozen, and sampling resumes without replaying completed evaluations. Across minWM-Wan Action2V and HY-WM1.5, the retargeting variant CST-R reduces rollback-relative LPIPS by 61.5% and 75.9% versus direct condition swapping, while the temporal-splicing variant CST-T reduces suffix LPIPS by 56.1% and 77.5% with 2.73x and 1.69x pixel-ready speedups over waiting.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

22

What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

Researchers diagnose conditional visual grounding failures in visuomotor imitation policies and show targeted interventions substantially improve distractor robustness.

The paper studies why ACT-based visuomotor imitation policies fail when visually similar distractor objects or receptacles are introduced, finding sensitivity depends on both distractor type and manipulation stage. Interventions including distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting improve target selection while preserving spatial control information, with gains in simulation and on a physical UR3e. The same failure pattern is confirmed in a pretrained vision-language-action policy on a state-conditioned medical instrument-handling task.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:27 UTC · 12d agoAI research

18

World in World: Explore the World with World Models

World in World is a training-free inference-time interface enabling camera-controlled rerendering, long-horizon revisiting, and motion transfer with frozen causal video world models.

The framework converts heterogeneous control evidence into camera- and time-labelled clean visual states that are read through the native self-attention of a frozen causal video model. Evidence includes source-video observations, target-view scene projections, geometry renderings for completing newly exposed regions, and retrieved generated states beyond the rolling cache. A correspondence router combines persistent point identities with geometry for token correspondences, while evidence-wise attention CFG independently regulates each auxiliary channel. The shared interface supports camera-controlled rerendering, long-horizon revisiting, and human-motion transfer without additional training.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

30

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled pure-autoregressive testbed shows task-specific validation losses rank image tokenizers differently, with I2T loss the most consistent signal.

Researchers built a controlled pure-autoregressive testbed and tracked task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, correlates consistently with both generation and visual understanding performance after supervised finetuning. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:57 UTC · 8d agoAI research1

30

Adobe is trying to make its AI generators idiot-proof in Premiere

Adobe adds in-timeline generative media to Premiere, letting editors generate video and audio clips using Firefly, Veo, Runway, Luma, and Kling models.

Adobe's new Generative Media tool lets Premiere editors highlight empty gaps in the timeline and generate context-aware, editable video, sound effects, music, and soundscapes without leaving the project. Editors can choose among underlying models including Adobe Firefly, Google Veo, Runway, Luma, and Kling. Beta AI audio tools can separate overlapping speakers and duck music under speech, and an AI Assistant is coming to After Effects for plain-language project commands.

The Verge · AI·Sep 8, 13:00 UTC · 8d agoAI industry

22

TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agents

TRACE, a training-free visual token pruning framework, cuts GUI agent inference latency and memory while keeping trajectory-wide visual evidence reusable.

TRACE is a training-free framework for trajectory-robust admission and coverage-aware evidence ordering that prunes high-resolution screenshot tokens accumulated in GUI agent trajectories. It ranks visual evidence using a query-independent layout-derived interaction prior combined with instruction relevance and feature novelty, and reserves part of the budget for native tokens distributed across the screen to repair spatial coverage. A monotone KV contraction incrementally compresses retired frames into compact session state, avoiding repeated visual encoding or pruning. Experiments across six GUI benchmarks and diverse models verify effectiveness under tight budgets, with source code to be released.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

28

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

Latent Interface Training improves robot foundation model generalization by constraining visual conditioning, boosting LIBERO-Plus success up to 10.7 points.

The paper identifies vision-action shortcuts where robot policies exploit task-irrelevant visual cues that fail under distribution shift. Latent Interface Training (LIT) first trains an action expert conditioned on language, robot state, and terminal SE(3) end-effector poses without images, then constrains visual input through a pose-supervised latent interface. Across four VLA and world-action architectures (Pi0.5, MolmoAct2, FAST-WAM, ImageWAM), LIT improves LIBERO-Plus success by 3.87-10.70 percentage points. Real-world tests show 13.30-16.70 percentage-point gains under unseen cameras, lighting, and distractors.

Hugging Face daily papers·Sep 10, 20:00 UTC · 5d agoAI research

32

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled autoregressive testbed shows validation losses must be analyzed per task, and image tokenizer choice affects joint multimodal text modeling.

Researchers built a pure-autoregressive testbed to study image tokenizers as the 'visual language' of unified multimodal models, tracking task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They found that losses exhibit distinct scaling behavior per task and rank tokenizers differently, and that I2T loss over a shared text vocabulary gives a more consistent loss–performance signal than T2I loss. Better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and tokenizer choice can affect text modeling under joint optimization. Case studies examine the discriminator, semantic supervision, and vocabulary size design axes.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research1

40

Context Engineering Inside the Harness: 4 Mechanisms That Beat Context Overflow and Goal Loss on Long-Horizon Tasks

Survey of four harness mechanisms—context budgeting, compaction, todo-state, and memory—that keep long-horizon LLM agents on task across 200+ tool calls.

The article details how agent harnesses, not larger context windows, solve context overflow and goal loss on long-horizon tasks, citing Chroma's Context Rot report showing 18 LLMs (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) degrade on long inputs. Concrete implementations include LangChain Deep Agents offloading tool responses over 20,000 tokens to the filesystem and truncating old tool calls at 85% window usage, and Claude Code capping auto memory at 25KB while re-reading the 5 most recently modified files after compaction. OpenAI's Responses API now offers server-side compaction via context_management with a standalone /responses/compact endpoint, which Codex uses for long-running coding tasks. Manus reports a roughly 100:1 input-to-output token ratio per ~50-tool-call task, motivating todo.md state recitation to prevent goal drift.

MarkTechPost·Sep 13, 05:56 UTC · 3d agoAI research1

52

VoT: Vision-of-Thought for Unified Multimodal Representation Alignment

Researchers propose Vision-of-Thought (VoT), a discrete visual-planning token layer between VLMs and diffusion transformers improving text-to-image semantic alignment.

VoT introduces a discrete visual-thinking layer between vision-language models and diffusion transformers, letting the VLM act as a multimodal planner that emits tokens describing objects and layouts before pixel generation. A specialized VoT tokenizer is trained with VLM alignment, feature reconstruction, and vector-quantization losses. Experiments show improved semantic alignment and a structured, interpretable interface for controllable generation.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:53 UTC · 9d agoAI research

24

CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs

CoVeR, a training-free coverage-based token pruner, preserves 93.5% of VLM 3D-reasoning performance using only about 8% of visual tokens.

Researchers introduce CoVeR, a deterministic, training-free selector that chooses visual tokens to cover every region of a multi-view 3D scene using only token coordinates. Unlike learned-importance and voxelization pruners, it enforces an exact per-scene token budget, avoids saturation plateaus, and prevents near-duplicate selections. Experiments across four vision-language models show it surpasses prior state of the art by 3.9 percentage points on average across three 3D reasoning benchmarks.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

45

Reason Through the Latent! Making Latent Visual Reasoning Necessary

Researchers introduce CVRR, forcing multimodal models to rely on recurrent latent computation rather than accessible image tokens, validated via causal interventions and benchmarks.

The paper presents Causal Visual Recurrent Reasoning (CVRR), which makes recurrent hidden-state computation the required image-conditioned path for prediction in vision-language models. Before decoding, visual states and the original multimodal KV cache are removed so only the final recurrent state carries image information to the answer. CVRR retains strong performance on V*, MMVP, BLINK, and MME-RealWorld-Lite while comparable latent reasoners fail under the same constraint. Causal interventions show predictions remain sensitive to recurrent content and that persistent visual evidence causally revises the recurrent trajectory.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

25

TempCloze: Can Video-LLMs Identify the Missing Middle?

TempCloze benchmark tests Video-LLMs' temporal reasoning with 1,521 videos, finding temporal alignment is the primary failure mode across 31 models.

TempCloze is a video cloze benchmark in which models must identify the true missing middle clip given the beginning and ending clips, using 1,521 carefully filtered videos from seven sources, mostly long-take and egocentric footage. Distractors are constructed along three dimensions: Semantic, Alignment and Progression, with shared scenes and objects to reduce appearance cues. Evaluation of 10 proprietary and 21 open-source Video-LLMs found Alignment is the primary bottleneck, with models often recognizing plausible semantics and local event progression but struggling with temporal alignment.

Hugging Face daily papers·Aug 31, 20:00 UTC · 15d agoAI research

25

Show HN: MultiMatte, a Promptable Image Background Removal Model

Feyn releases MultiMatte, a promptable background-removal model fine-tuned from Meta's SAM 3 via LoRA, outputting alpha mattes that beat SAM 3 on segmentation benchmarks.

Feyn introduced MultiMatte, a promptable image background-removal model built on Meta's SAM 3 (860M parameters). It modifies only 19.49M parameters (2.27%) using a rank-16 LoRA adapter and replaces binary masks with alpha mattes to handle fuzzy boundaries like hair. On the DIS-VD benchmark it scores 0.901 S-measure versus SAM 3's 0.667, and it improves on SAM 3 across all twelve evaluated splits. Training used 19,953 images for 14,000 steps with focal and Dice loss, and the merged weights are available via the nobg library and a web demo.

Hacker News · security·Sep 10, 15:50 UTC · 6d agoModel releaseHN 21↑ · 4 comments1

45

Unfold The World: Factorize 4D Properties in Reinforcing Spatial Reasoning

FactoSR factorizes 4D spatial reasoning into XY, Z, and T reinforcement-learning sub-objectives, boosting VLM performance on VSI-Bench by 5.9% and All-Angles-Bench by 4.5%.

Researchers present FactoSR, a factorized reinforcement learning framework that decomposes world-consistent reasoning into planar correspondence, depth consistency, and temporal reversibility sub-objectives. Optimizing these verifiable constraints turns the ill-posed projection recovery problem into tangible reasoning steps. Evaluations show gains of 5.9% on VSI-Bench and 4.5% on All-Angles-Bench for 3D and 4D reasoning, arguing VLMs' spatial bottleneck stems from training on 2D projections versus latent 3D geometry and temporal continuity.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

26

ENEAS: Embedding-guided Neural Ensemble for Adaptive Segmentation

ENEAS adds text prompting and semantic verification to video segmentation to keep tracking targets through occlusion and reject lookalike distractors.

ENEAS is a unified text-promptable method for instance tracking and open-concept semantic discovery in video, designed to fix temporal hallucinations, spatial fragmentation, and semantic misclassification seen in SAM 3-class foundation models. It extends the geometrically robust SeC architecture with a text-prompting adapter and temporal memory, and uses a verification layer combining fast visual embedding matching with conditional VLM refinement for ambiguous candidates. It targets 3D reconstruction pipelines where a single misclassified distractor corrupts the asset. Code and models are open-sourced.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

32