Search: “visual-grounding”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control

PhysStream enables mid-generation interactive control of physics-grounded video via structured scene memory and velocity-increment signals, reducing motion distribution distance 33%.

PhysStream is an autoregressive physics-grounded image-to-video model that maintains structured scene memory—positional maps and object tracking maps derived online from previously generated frames—and accepts fine-grained motion control via sparse velocity-increment signals encoding physical quantities. Training runs in two stages: a bidirectional model finetuned with motion-control conditioning, then a causal autoregressive model with structured scene memory. It supports interactive mid-generation control over multi-object tabletop rigid-body scenes, reducing motion distribution distance (FVMD) by 33% and trajectory error by 12% over the strongest baselines. Human evaluators preferred it in over 85% of in-the-wild comparisons.

Hugging Face daily papers·updated Sep 15, 17:55 UTC · 19h ago·first Sep 14, 20:00 UTC · 1d agoAI research 2 sources

28

RelightFormer: Feed-forward Generative Transformer for Multiview Object Relighting

RelightFormer is a feed-forward generative transformer for photorealistic single- and multi-view object relighting, trained on a 90K-object dataset.

Researchers introduce RelightFormer, a feed-forward generative transformer adapted from a video foundation model that performs direct image relighting without explicit intrinsic property estimation. The architecture injects target environment maps via a latent illumination module with cross-attention and uses permutation-invariant positional encodings for unordered multi-view inputs. Training relies on the newly constructed Laval Objaverse Dataset (LOD) with 90K objects and 39K unique illuminations, and the model shows state-of-the-art quality with strong zero-shot generalization across single-view, multi-view, and novel-view relighting.

Hugging Face daily papers·Sep 6, 20:00 UTC · 9d agoAI research

38

What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

Researchers diagnose conditional visual grounding failures in visuomotor imitation policies and show targeted interventions substantially improve distractor robustness.

The paper studies why ACT-based visuomotor imitation policies fail when visually similar distractor objects or receptacles are introduced, finding sensitivity depends on both distractor type and manipulation stage. Interventions including distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting improve target selection while preserving spatial control information, with gains in simulation and on a physical UR3e. The same failure pattern is confirmed in a pretrained vision-language-action policy on a state-conditioned medical instrument-handling task.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:27 UTC · 11d agoAI research

18

AdaptVPR: Route-Aware Hard Positive Generation for Robust Visual Place Recognition

AdaptVPR generates route-aware synthetic hard positives for visual place recognition, releasing the 160K-image AdaptCities dataset with R@1 gains up to 9.2% under domain shift.

AdaptVPR is a generative augmentation framework that creates same-place hard positives under illumination, weather, seasonal, and dynamic-occlusion shifts for robust visual place recognition training. A vision-language model parses scene attributes and estimates editability, while a rule-based scheduler routes generation through global appearance, local occlusion, or dual perturbation routes with geometric-consistency verification. The resulting AdaptCities dataset contains 160K verified synthetic hard positives, and experiments show R@1 gains up to 9.2% across VPR baselines and backbones. Code and data are publicly released on GitHub.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

22

RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives

RenderFormer-V2 is a transformer-based neural renderer handling caustics, volumetric scattering and out-of-distribution materials without per-scene training or specialized code.

RenderFormer-V2 is a learned transformer-based neural rendering model that models global light transport as a sequence-to-sequence transformation, handling caustics, volumetric scattering, environment lighting, textured and displaced surfaces and out-of-distribution materials. It uses a two-stage process: a view-independent stage resolving primitive-to-primitive transport, and a view-dependent stage converting the neural scene representation into pixels. Improvements include combined windowed-attention with a rendering-informed attention sink for scalability, support for heterogeneous primitives like environment maps and participating media, and a surface-reflectance-independent neural material encoding, validated across diverse scenes with extensive ablations.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research

22

Hierarchical NeRF with JAX3D for Volumetric Rendering, Novel-View Synthesis, and 3D Reconstruction

MarkTechPost tutorial implements a hierarchical NeRF in JAX using jax3d volume-rendering primitives for novel-view synthesis and 3D reconstruction.

The tutorial builds an end-to-end hierarchical Neural Radiance Field using JAX, Flax, Optax, and jax3d's volume-rendering functions (sample_along_rays, volume_rendering, sample_piecewise_constant_pdf). It implements positional encoding, skip connections, separate coarse and fine networks, and view-direction conditioning with hierarchical importance sampling. Training uses JAX JIT compilation, Adam optimization, exponential learning-rate decay, and gradient clipping. Evaluation covers PSNR, depth and opacity visualization, 360-degree rendering, and marching-cubes geometry extraction.

MarkTechPost·Sep 13, 19:46 UTC · 2d agoAI research

18

TRACE: Trajectory-robust Admission with Evidence Ordering for Efficient GUI Agents

TRACE, a training-free visual token pruning framework, cuts GUI agent inference latency and memory while keeping trajectory-wide visual evidence reusable.

TRACE is a training-free framework for trajectory-robust admission and coverage-aware evidence ordering that prunes high-resolution screenshot tokens accumulated in GUI agent trajectories. It ranks visual evidence using a query-independent layout-derived interaction prior combined with instruction relevance and feature novelty, and reserves part of the budget for native tokens distributed across the screen to repair spatial coverage. A monotone KV contraction incrementally compresses retired frames into compact session state, avoiding repeated visual encoding or pruning. Experiments across six GUI benchmarks and diverse models verify effectiveness under tight budgets, with source code to be released.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research

28

Reason Through the Latent! Making Latent Visual Reasoning Necessary

Researchers introduce CVRR, forcing multimodal models to rely on recurrent latent computation rather than accessible image tokens, validated via causal interventions and benchmarks.

The paper presents Causal Visual Recurrent Reasoning (CVRR), which makes recurrent hidden-state computation the required image-conditioned path for prediction in vision-language models. Before decoding, visual states and the original multimodal KV cache are removed so only the final recurrent state carries image information to the answer. CVRR retains strong performance on V*, MMVP, BLINK, and MME-RealWorld-Lite while comparable latent reasoners fail under the same constraint. Causal interventions show predictions remain sensitive to recurrent content and that persistent visual evidence causally revises the recurrent trajectory.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

25

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled pure-autoregressive testbed shows task-specific validation losses rank image tokenizers differently, with I2T loss the most consistent signal.

Researchers built a controlled pure-autoregressive testbed and tracked task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, correlates consistently with both generation and visual understanding performance after supervised finetuning. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:57 UTC · 7d agoAI research1

30

Do Input-Level Defenses Transfer to Observation-Level Attacks on VideoLLMs?

A systematic study shows input-level adversarial defenses provide inconsistent, often near-zero protection against observation-level attacks on video LLMs.

Researchers introduce DefTEval, a controlled framework testing eleven input-level defenses against five attack types across five video LLMs. Harmful-content detection rates are frequently near zero, and defenses fail even when attacks embed harmful signals in every sampled frame. Token compression discards localized safety features and modality fusion down-weights weakened visual signals, with defense outcomes dominated by model architecture rather than the defense method.

arXiv cs.CR·Sep 8, 06:59 UTC · 8d agoAI safety & security

40

Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model

Study shows visually grounded token embeddings in a small masked LM persist through training and improve object-property knowledge, but escape standard BabyLM benchmarks.

The paper implements ostensive definition for a small DeBERTa masked language model trained on 10M words, seeding visually grounded tokens with embeddings derived from labeled image regions before training. Visual initialization leaves a persistent, seed-replicated advantage on object-property knowledge (COMPS) and a corpus-tailored Visual-Property Swap benchmark covering color, material, size, and shape, but has no effect on most BabyLM grammar benchmarks. Synthetic grounding of previously unseeded words causally transfers the advantage to exactly those words.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:43 UTC · 5d agoAI research

20

TransNormal-2: Geometry-Grounded Rectified Flow with Edge-Aware Decoding for Precise Normal Estimation

TransNormal-2 improves monocular surface-normal estimation by fixing VAE edge degradation with geometry-aware losses and refinement, matching MoGe-2 with 1.4% of annotations.

TransNormal-2 is a FLUX.2-based rectified-flow framework for monocular surface-normal estimation with single-step deterministic inference. The authors quantify that VAE 8x spatial compression introduces 1.3-8.5 degrees of mean angular error even on ground-truth normals, with edge error up to 2.8x the global error. The method adds geometry-aware pixel-space losses and an RGB-guided Geometric Refinement Module to correct boundary-localized decoding errors. It matches or exceeds MoGe-2 on all eight reported metrics using only 1.4% as many task-specific annotations, and cuts transparent-object MAE by 4.2 degrees on ClearGrasp and 3.1 degrees on ClearPose.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

20

Ambient @ EgoProactive 2026 : Proactive Egocentric Assistance with Visually Grounded Supervision

ECCV 2026 challenge winner reformulates egocentric intervention timing as single-token classification, boosting macro-F1 by 0.249 over free-form generation.

The paper describes the winning submission to the EgoProactive track of the ECCV 2026 Wearable AI Challenge, ranking first in the large-model division and second in the <=2B division. The method reformulates intervention timing as single-token yes/no classification, improving macro-F1 by 0.249 and G-mean by 0.30 over free-form generation. Supervision generated by a tool-calling video agent transferred better than a narration-only dataset that was four times larger and ten times cheaper, suggesting visual grounding matters more than annotation volume.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

22

VoT: Vision-of-Thought for Unified Multimodal Representation Alignment

Researchers propose Vision-of-Thought (VoT), a discrete visual-planning token layer between VLMs and diffusion transformers improving text-to-image semantic alignment.

VoT introduces a discrete visual-thinking layer between vision-language models and diffusion transformers, letting the VLM act as a multimodal planner that emits tokens describing objects and layouts before pixel generation. A specialized VoT tokenizer is trained with VLM alignment, feature reconstruction, and vector-quantization losses. Experiments show improved semantic alignment and a structured, interpretable interface for controllable generation.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:53 UTC · 8d agoAI research

24

CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs

CoVeR, a training-free coverage-based token pruner, preserves 93.5% of VLM 3D-reasoning performance using only about 8% of visual tokens.

Researchers introduce CoVeR, a deterministic, training-free selector that chooses visual tokens to cover every region of a multi-view 3D scene using only token coordinates. Unlike learned-importance and voxelization pruners, it enforces an exact per-scene token budget, avoids saturation plateaus, and prevents near-duplicate selections. Experiments across four vision-language models show it surpasses prior state of the art by 3.9 percentage points on average across three 3D reasoning benchmarks.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

45

Show HN: LLM Attention Visualization

A developer released a browser-based tool that visualizes which past tokens influence each LLM output token using aggregated, value-weighted attention scores.

A Show HN project presents a React application built on Transformers.js that renders per-token attention influence by aggregating attention weights scaled by value-vector magnitudes across all attention heads and layers. To expose internal tensors, the author instrumented the ONNX computation graph, hosted a modified model on Hugging Face, and pre-generated prompts to avoid long model downloads in the browser. Demos with a 600-million-parameter model show how verbatim copying draws heavily on source tokens and how single outputs blend information from multiple phrases.

Hacker News · AI·Sep 8, 16:59 UTC · 7d agoAI tools & infraHN 41↑ · 9 comments

28

Multi-Grid Post-Training for Long-Form Multi-Shot Video Generation

MovieGrid arranges long videos on spatial grids during post-training, generating 6.05x more shots than temporal packing with state-of-the-art cross-shot consistency.

MovieGrid is a multi-grid post-training paradigm that decomposes long videos into temporally ordered chunks arranged on a spatial grid for joint modeling, enabling cross-chunk information exchange. The authors build the Multi-Grid Long Video (MGLV) dataset from 1,000 long-form videos, producing 54K grid videos paired with character-aware story prompts. Under the same token budget, MovieGrid generates 6.05x more shots than Temporal Packing in a 1,616-frame video. It achieves state-of-the-art intra-shot consistency of 0.9131 versus 0.8086 for HoloCine and inter-shot consistency of 0.5914 versus 0.5384 for StoryMem.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

20

Agentic Visual Generation: From Generative Models to Agentic Control

Researchers propose an L0-L4 control taxonomy for agentic visual generation, classifying controllers from fixed conditioning to experience-adaptive decision-making.

This paper proposes a taxonomy for agentic visual generation organized by what the controller can directly control in the generation process, rather than by planning depth, tool count, or model size. Levels range from L1 Conditioning Control through L2 Execution Control, L3 Outcome-Adaptive Control, and L4 Experience-Adaptive Control, with L0 Fixed Support denoting systems without a deployed decision-making controller. The framework is applied across image, video, editing, 3D, world, slide, and user-interface generation to map how controller capabilities and mechanisms have evolved across the field.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

22

Learning 3D Editing without Paired Supervision via Generative Prior Distillation

New framework distills 2D editing and VLM priors into a feed-forward 3D editing model without paired 3D training data.

The method, PriorEdit3D, learns feed-forward instruction-guided 3D editing by distilling knowledge from foundation models instead of using ground-truth 3D pairs. Through a differentiable rendering pipeline it supervises a 2D visual prior from an image editing model at the main view and a Vision-Language Model semantic prior at novel views for instruction fidelity and identity preservation. A 3D-aware Distribution Matching regularization constrains outputs to the manifold of realistic 3D assets defined by a pretrained image-to-3D teacher. Experiments report superior instruction fidelity and cross-view consistency over state-of-the-art baselines, with code released on GitHub.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research

30

Recursive Code World Models: Building Complex Worlds through Recursive Scene Programs

RCWM reconstructs complex 3D worlds as executable code from a single image using recursive scene programs with global-local-global solver recursion.

The paper introduces Recursive Code World Models, coupling a Recursive Scene Program representation with a recursive construction solver for image-to-3D-world reconstruction. Each solver call establishes the whole scene, recursively reconstructs unresolved parts, and revisits the whole to refine composition, while a vision-language coding agent compares reference images with scene renders to guide refinement. RCWM outperforms prior code-based image-to-scene reconstruction methods, and ablations show deeper recursive calls improve fine-scale reconstruction.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research1

33

3D Point Splatting for mmWave Radar Novel View Synthesis

Researchers propose 3DPS, a differentiable point renderer for mmWave radar novel view synthesis that outperforms optical-NVS baselines by 1.7x-5.2x.

The paper introduces 3D Point Splatting (3DPS), the first differentiable point renderer for radar, derived from the solid-angle form of the radar equation with ITU-R P.2040 material models and complex phasor splatting. On six outdoor ColoRadar scenes it reaches 0.587 mean Pearson correlation on held-out range-azimuth images, between 1.7x and 5.2x the RadarSplat, Radar Fields, and DART baselines. The same optimized scene produces ADC, complex range profile, and RA outputs via standard FFT pipelines, and training takes about 3 minutes per scene on an RTX 4090.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:52 UTC · 5d agoAI research

28

Reflection-aware Generative Novel View Synthesis

Ref-GeNVS is a training-free method for reflection-consistent generative novel view synthesis that treats mirror images as two complementary views.

An arXiv paper proposes Ref-GeNVS, a training-free approach to generative novel view synthesis in scenes containing mirrors. It estimates the mirror plane, reflects camera poses to create virtual views, and applies mirror-gated attention plus reflection injection within a multi-view diffusion model. On synthetic and real mirror scenes, Ref-GeNVS outperforms recent generative NVS methods while requiring no fine-tuning.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:33 UTC · 11d agoAI research

26

SceneMosaic: Efficient and Diverse Simulation-Ready Scene Generation via Hybrid Agentic Layout Evolution

SceneMosaic combines image-based 3D priors with VLM agent refinement to generate diverse, simulation-ready indoor scenes 24x faster than agentic baselines.

SceneMosaic is a hybrid framework that takes an initial candidate from a learned image-to-3D prior and evolves it with VLM agents for efficiency and physical validity. It decomposes scenes into independent local units, evolves each separately, and composes the global scene via Cartesian product. On SceneEval-100 it matches the strongest agentic baseline in semantic layout quality with a 24x speedup, substantially reduces physical violations, and receives the highest human ratings. Code is publicly available.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research

25

Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

Marigold V2 adapts diffusion transformers for monocular depth estimation, improving AbsRel 16-26% over the previous best on KITTI and ETH3D.

Huawei's Bayer lab revisits the Marigold approach to repurpose image generation and editing models built on the diffusion transformer (DiT) architecture into monocular depth estimators. The recipes target single-step inference from pretrained multi-step flow-matching models, with remedies including alignment to ground-truth semantic features and a two-stage fine-tuning protocol using a Sinkhorn-based loss. The resulting model produces crisper depth maps that generalize out-of-distribution and also achieves state-of-the-art results on surface normals estimation and intrinsic image decomposition.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

42

OracleZoom: On-Policy Self-Distillation Inspired Reference-Constrained Recursive Image Super Resolution

OracleZoom enables recursive extreme-scale image super-resolution via reference-constrained on-policy distillation, reducing hallucinations at deep zoom scales.

OracleZoom tackles recursive super-resolution, where repeated feeding of predictions back into the same model leaves deeper-scale outputs unsupervised as required source resolution grows geometrically. The framework trains on its own trajectory while carrying the last ground-truth evidence beyond the supervision boundary, combining direct and cross-scale supervision, a no-reference quality objective, a KL-constrained pretrained latent prior, and EMA consistency. Across seven datasets it achieves state-of-the-art SR quality across zoom scales, averaging 0.713 CLIPIQA with larger gains at deeper scales and significantly reduced hallucinations. Code, data, and models are publicly released.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

20

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

Latent Interface Training improves robot foundation model generalization by constraining visual conditioning, boosting LIBERO-Plus success up to 10.7 points.

The paper identifies vision-action shortcuts where robot policies exploit task-irrelevant visual cues that fail under distribution shift. Latent Interface Training (LIT) first trains an action expert conditioned on language, robot state, and terminal SE(3) end-effector poses without images, then constrains visual input through a pose-supervised latent interface. Across four VLA and world-action architectures (Pi0.5, MolmoAct2, FAST-WAM, ImageWAM), LIT improves LIBERO-Plus success by 3.87-10.70 percentage points. Real-world tests show 13.30-16.70 percentage-point gains under unseen cameras, lighting, and distractors.

Hugging Face daily papers·Sep 10, 20:00 UTC · 5d agoAI research

32

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled autoregressive testbed shows validation losses must be analyzed per task, and image tokenizer choice affects joint multimodal text modeling.

Researchers built a pure-autoregressive testbed to study image tokenizers as the 'visual language' of unified multimodal models, tracking task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They found that losses exhibit distinct scaling behavior per task and rank tokenizers differently, and that I2T loss over a shared text vocabulary gives a more consistent loss–performance signal than T2I loss. Better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and tokenizer choice can affect text modeling under joint optimization. Case studies examine the discriminator, semantic supervision, and vocabulary size design axes.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research1

40

Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout

Researchers propose Mask Forcing, a dual-noise masking rollout that mitigates mode collapse in autoregressive video diffusion distillation.

The paper targets over-saturation and over-smoothing in autoregressive video diffusion models distilled via Distribution Matching Distillation, attributed to reverse-KL mode-seeking behavior. Mask Forcing perturbs the student self-rollout with random masks along spatial and temporal axes, injecting cleaner tokens that act as denoising guidance for noisier tokens. Experiments show improved visual quality across multiple distillation methods without using real video data or extra post-training stages.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

22

AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video

AlayaVista is a camera-controllable streaming video world model that decouples panoramic scene evolution from perspective synthesis, trained on a 1,318-hour 4K dataset.

AlayaVista builds a 360-degree scene prior from a single perspective image, evolves it as a camera-conditioned panoramic latent state, and maps it to perspective video via a latent viewport renderer plus a perspective refiner. Chunk-autoregressive generation and few-step distillation enable efficient streaming. The authors introduce MUGEN, a real-world panoramic video dataset with 1,318 hours of at-least-4K video and rich semantic and geometric annotations.

Hugging Face daily papers·Sep 12, 20:00 UTC · 3d agoAI research

45