Search: “attention-maps”

Smart search ranks by meaning as well as keywords (one row per story, last 45 days).

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

SAS trains attention sparsification end-to-end with the language modeling loss, beating sparse attention baselines especially under tight context budgets.

Simple Attention Sparsification (SAS) injects the selector's continuous scores into attention logits in log form inside the softmax, letting gradients from the language modeling loss directly update the ranking of context units. The method uses normalized softmax gates calibrated against the current block and a memory-efficient Triton kernel integrated into FlashAttention-style computation. Across reasoning, long-context, and agentic tasks, SAS consistently outperforms trainable sparse attention baselines across budgets, with the largest gains under tight attention budgets.

Hugging Face daily papers·updated Sep 11, 17:58 UTC · 5d ago·first Sep 10, 20:00 UTC · 5d agoAI research 2 sources1

30

CodeTD: Topology of Attention Detects Hallucinations in Code LLMs

CodeTD detects hallucinations in code LLMs before execution by analyzing topological patterns of attention maps, outperforming recent baselines.

CodeTD applies topological data analysis (TDA) to code LLM attention maps to quantify prompt-generation mismatch as a pre-execution correctness signal. Experiments cover HumanEval, MBPP, BigCodeBench, and MultiPL-E across 5 programming languages and 10 code LLMs up to 34B parameters. The method outperforms recent baselines and transfers between coding benchmarks, helping catch code that fails the task or embeds security vulnerabilities.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:26 UTC · 9d agoAI research1

24

Memory as Plans: World-Action Modeling with Memory-Grounded Planning

Researchers introduce MaP-WAM, decomposing memory-dependent robot manipulation into memory-grounded planning and plan-conditioned execution, achieving 83.3% on RMBench and 78% on real robots.

MaP-WAM converts long-term multimodal episodic memory — segment records with language instructions and sparse visual context — into compact plans of next-segment language goals and visual guidance. A World-Action-Progress model jointly predicts action chunks and execution progress, calibrating predictions via plan-observation alignment for adaptive segment transitions and closed-loop context updates. Structured attention keeps the executor context length fixed and enables key-value caching, yielding state-of-the-art 83.3% success on RMBench, 78.0% on real-robot tasks, and roughly constant inference latency as task history grows.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

32

Show HN: LLM Attention Visualization

A developer released a browser-based tool that visualizes which past tokens influence each LLM output token using aggregated, value-weighted attention scores.

A Show HN project presents a React application built on Transformers.js that renders per-token attention influence by aggregating attention weights scaled by value-vector magnitudes across all attention heads and layers. To expose internal tensors, the author instrumented the ONNX computation graph, hosted a modified model on Hugging Face, and pre-generated prompts to avoid long model downloads in the browser. Demos with a 600-million-parameter model show how verbatim copying draws heavily on source tokens and how single outputs blend information from multiple phrases.

Hacker News · AI·Sep 8, 16:59 UTC · 8d agoAI tools & infraHN 41↑ · 9 comments

28

It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention

Study shows attention sinks and massive activations stem from causal-mask self-concentration and value-non-mixing rather than RoPE, informing quantization work.

The paper analyzes why attention sinks and massive activations emerge at initial sequence positions regardless of which token occupies them. Experiments attribute both phenomena to self-concentration of attention induced by the causal mask and the subsequent value-non-mixing in attention outputs. The findings provide empirical evidence on LLM internal dynamics and may inform low-bit quantization strategies, which massive activations currently complicate.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:32 UTC · 8d agoAI research

32

Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction

Grouped Value Attention stores grouped values and reconstructs content keys via a learned linear map, cutting KV-cache size about 45-47% versus GQA.

GVA stores only grouped values and reconstructs content keys with a learned linear map absorbed into the query at decode time, while a small shared decoupled RoPE channel preserves positional information via a separately cached positional key. At 350M parameters trained on 30B FineWeb-Edu tokens, the 16-dimensional positional variant scores 44.18 average accuracy across five tasks versus 44.36 for GQA and 43.88 for MLA. Custom decoding kernels are in development with an open-source release planned.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research

42

The Attention Triangle in Audio-Video Models

Researchers analyze the 'attention triangle' in audio-video diffusion models, showing bias-driven cross-attention routing causes semantic leakage and proposing inference-time interventions that improve grounding.

A study probes the three cross-attention edges linking text, audio, and video streams in audio-video diffusion models. It finds the audio-video edge is bidirectional and shaped by parameter-encoded biases, so prompts in tension with learned priors can be overridden, producing visually canonical but incorrect outputs. Attention-derived signals are used as diagnostics and to guide inference-time interventions that improve cross-modal semantic grounding while preserving generation quality.

Hugging Face daily papers·Sep 2, 20:00 UTC · 13d agoAI research

24

Attention-DP3: Spatially Object-aware 3D Diffusion Policy via Geometry-aligned Attentional Conditioning

Attention-DP3 adds spatially object-aware attentional conditioning to 3D diffusion policies, improving robotic manipulation by up to 31% under heavy clutter.

Attention-DP3 injects object-level geometric cues into the unchanged DP3 diffusion policy via Tri-field Attentional Conditioning, using targetness, intra-target saliency, and backgroundness fields. Open-vocabulary 2D segmentation masks are lifted to 3D with calibrated camera geometry to build object-centric priors. Experiments on Adroit, DexArt, MetaWorld, and a real-world SO101 platform show state-of-the-art results, outperforming DP3 by up to 31% under heavy distractor clutter; the code is publicly available on GitHub.

Hugging Face daily papers·Sep 9, 20:00 UTC · 6d agoAI research

15

Quantifying the Engagement Trap: Impact of Short-form Video Recommender Systems on Users with ADHD

A 302-participant study finds engagement-optimized short-form video recommenders cause disproportionate time blindness and distress for users with ADHD.

Researchers ran a stratified Prolific study with 302 participants comparing short-form video recommendation experiences with and without ADHD. Participants with ADHD reported significantly higher time blindness, post-usage regret, and emotional distress despite perceiving recommendations as similarly relevant. The paper proposes neurodiversity-aware, human-centered design interventions to mitigate these algorithmic harms.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:38 UTC · 9d agoAI research

18

What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies

Researchers diagnose conditional visual grounding failures in visuomotor imitation policies and show targeted interventions substantially improve distractor robustness.

The paper studies why ACT-based visuomotor imitation policies fail when visually similar distractor objects or receptacles are introduced, finding sensitivity depends on both distractor type and manipulation stage. Interventions including distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting improve target selection while preserving spatial control information, with gains in simulation and on a physical UR3e. The same failure pattern is confirmed in a pretrained vision-language-action policy on a state-conditioned medical instrument-handling task.

arXiv cs.AI / cs.LG / cs.CL·Sep 4, 17:27 UTC · 12d agoAI research

18

Kalman Delta Networks: Uncertainty-aware Associative Memory

Kalman Delta Networks add uncertainty tracking to linear-attention associative memory, improving perplexity and downstream accuracy at 750M and 1.3B scales.

Kalman Delta Networks reformulate recurrent associative memory in linear-attention models as a linear-Gaussian state-space model, allowing the Kalman gain to weight each residual write by accumulated evidence and observation reliability; Delta-rule updates emerge as a special case lacking covariance tracking. Two scan-compatible approximations, Diagonal KDN (online mean-field variational inference) and Isotropic KDN (one uncertainty scalar per head), produce Mobius-map uncertainty recurrences enabling associative scans with logarithmic parallel depth. Controlled pretraining at 750M and 1.3B parameters consistently improves perplexity and mean downstream accuracy over state-of-the-art linear-attention models.

arXiv cs.AI / cs.LG / cs.CL·Sep 7, 17:53 UTC · 9d agoAI research1

32

Kalman Delta Networks: Uncertainty-aware Associative Memory

Researchers propose Kalman Delta Networks, adding Kalman-filter uncertainty tracking to delta-rule linear attention, improving perplexity and accuracy at 750M and 1.3B parameters.

The paper introduces Kalman Delta Networks (KDNs), which reformulate recurrent associative memory in linear-attention models as a linear-Gaussian state-space model where the Kalman gain weights each write by accumulated evidence and observation reliability. Two scan-compatible approximations, Diagonal KDN via online mean-field variational inference and Isotropic KDN with a single uncertainty scalar per head, enable associative scans with logarithmic parallel depth. Delta-rule updates are shown to be a special case of this formulation. KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention baselines in controlled pretraining at 750M and 1.3B parameters.

Hugging Face daily papers·Sep 6, 20:00 UTC · 9d agoAI research

30

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled pure-autoregressive testbed shows task-specific validation losses rank image tokenizers differently, with I2T loss the most consistent signal.

Researchers built a controlled pure-autoregressive testbed and tracked task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They find losses should be analyzed per task because they exhibit distinct scaling behavior and rank tokenizers differently, and that the loss-performance relationship depends on the predicted token space. I2T loss, computed over a shared text vocabulary, correlates consistently with both generation and visual understanding performance after supervised finetuning. Case studies revisit the discriminator, semantic supervision, and vocabulary size as tokenizer design axes.

arXiv cs.AI / cs.LG / cs.CL·Sep 8, 17:57 UTC · 8d agoAI research1

30

Attention Quantization for Tabular Foundation Models

FP8 quantization of attention queries, keys, and values speeds tabular foundation model inference up to 1.7x with no accuracy loss.

The paper develops an FP8 quantization strategy targeting attention calculations (queries, keys, values) in tabular foundation models, arguing attention matters more than weight or KV cache quantization given their differing size and serving patterns versus LLMs. Aligning quantization error between test rows and training rows proves crucial, since misalignment causes drastic accuracy drops. A Triton kernel using explicit FP8 matrix multiplication achieves up to 1.7x speedup over regular 16-bit kernels, with no relevant accuracy loss on TabPFN-v3 and TabICLv2 across TabArena and BeyondArena benchmarks.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 16:31 UTC · 5d agoAI research1

28

The Semantic Bottleneck: Leveraging Semantic Representations for Non-Invasive Speech Decoding

Researchers introduced Brain2Semantics2Text, decoding sentence meaning from non-invasive MEG brain recordings via a semantic bottleneck, improving on prior Brain2Text methods.

The paper proposes Brain2Semantics2Text, a non-invasive speech decoding method that maps sentence-level magnetoencephalography (MEG) responses into a semantic embedding space and inverts those embeddings into natural language. Motivated by evidence that high-level semantic representations are distributed across cortex and evolve on slower timescales, the approach targets meaning rather than phonemes or words, avoiding the need for word-level alignment. The authors report improved sentence-level results compared to prior non-invasive Brain2Text methods despite the low signal-to-noise ratio of neural recordings.

Hugging Face daily papers·Sep 8, 20:00 UTC · 7d agoAI research2

25

Rethinking Heterogeneous System Disaggregation for Subquadratic Attention

SQD disaggregates LLM inference by quadratic versus subquadratic attention layers, improving energy efficiency up to 56% on heterogeneous systems versus GPU-only baselines.

SQD (SubQuadratic Disaggregation) splits decode not by operator type but by quadratic versus subquadratic attention, matching their distinct arithmetic intensity and memory footprints. For sparse attention LLMs it separates top-k selection (requiring full KV indexing) from top-k attention plus FFN; for linear and sliding-window models it separates dense attention layers from subquadratic layers plus FFN. On an adjusted 8xB200 heterogeneous proxy, tokens-per-joule improves 53% on GLM 5.2, 31% on Nemotron 3 Ultra, and 56% on Gemma 4 31B. A Rubin plus LPX analytical model shows 1.2x-1.5x tighter achievable latencies and up to 3.6x higher throughput versus attention-FFN disaggregation.

arXiv cs.AI / cs.LG / cs.CL·Sep 11, 17:55 UTC · 5d agoAI research

28

Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs

Study shows LLM reasoning operations like planning and deduction are geometrically separable in hidden states, with separability peaking in middle layers.

Researchers investigate whether functional reasoning operations — problem formulation, goal decomposition, deduction — have corresponding geometric structure in LLM hidden representations. They find operations are separable in held-out representations with separability peaking in middle layers, ruling out lexical and positional confounds; token-wise operation alignment becomes more distributed across layers, and identical surface tokens are represented differently depending on their surrounding chunk. Attention-masking interventions show chunk-onset operation-aligned representations depend on preceding reasoning context; code is released on GitHub (naver-ai/beneath-cot).

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research1

40

The Router Within: Eliciting Native Skill Routing from a Frozen LLM

Gavel reads native skill-routing signals from a frozen LLM's forward passes with two linear maps, beating retrieve-and-rerank pipelines by up to 21.9 points on Qwen3-32B.

Gavel (Glance And Verdict from a frozen LLM) elicits skill routing from a frozen agent LLM using two trained linear maps that read mid-layer states, keeping all skill text out of context. A glance step scores the full library against compact per-skill banks built in one forward pass at installation; a verdict step resumes shortlisted skills' forward passes and fuses likelihood and yes/no judgments as a product of experts. It transfers zero-shot to three public benchmarks plus SkillTraj, a new benchmark of 372 simulated agent trajectories. On Qwen3-32B it beats progressive disclosure and retrieve-and-rerank pipelines adding 1.2B–16B external parameters by up to 13.4 points on written tasks and 21.9 when skills are needed mid-rollout.

Hugging Face daily papers·updated Sep 14, 17:58 UTC · 2d ago·first Sep 13, 20:00 UTC · 2d agoAI research 2 sources

40

I-have-ADHD: A skill to stop your coding agent from burying the answer

Developer releases I-have-ADHD, an open-source skill that stops coding agents from burying direct answers under verbose output.

A GitHub project titled I-have-ADHD packages a 'skill' intended to make AI coding agents answer directly instead of hiding the answer inside lengthy output. It gained traction on Hacker News with 45 points and 42 comments. It targets prompt/agent configuration rather than introducing a new model or infrastructure component.

Hacker News · AI·Sep 8, 14:13 UTC · 8d agoAI tools & infraHN 45↑ · 42 comments1

28

Convergent Emergence of In-Context Learning Across Modalities

Controlled experiments show few-shot in-context learning emerges across six modalities including language, genomes, images, and proteins, partially supporting a convergence hypothesis.

The paper tests the Convergent Emergence Hypothesis: that few-shot in-context learning, when it emerges, shares a common cross-modality difficulty profile. A controlled framework instantiated the same task suite across six modalities: language, genome, integer sequences, time series, images, and proteins. Paired-mapping ICL emerged in all six modalities, surpassed controlled baselines, and showed correlated per-task effects in five of them, providing partial support for the hypothesis.

Hugging Face daily papers·Sep 11, 20:00 UTC · 4d agoAI research

30

Qwen-Drive 1.0 tells you why it brakes, just don't expect the explanation to match the maneuver

Alibaba's Qwen-Drive 1.0 adds 3D perception and planning modules to Qwen3.5-4B for driving tasks, though explanations often mismatch maneuvers.

Qwen-Drive 1.0, built on Qwen3.5-4B, combines spatial perception, traffic question answering, and route planning in one vision-language model, adding a bird's-eye-view perception module and a Planning Expert trained via staged fine-tuning and reinforcement learning. The paper finds text-image models do not inherently grasp 3D space; spatial accuracy only improved when the base vision-language model itself was trained on spatial tasks, while avoiding catastrophic forgetting of general knowledge. The cut reinforcement learning-trained version halved road-departure rate in simulation from 24% to 12%, and the model beats specialized driving models in most of Qwen's benchmarks, but its explanations sometimes conflate causes like distant red lights and crossing children, and results partly rest on self-designed tests. The work follows prior findings from PaLM-E and a UC Santa Cruz adversarial sign attack on DriveLM showing VLM driving models' reasoning and spatial gaps.

The Decoder·Sep 7, 12:15 UTC · 9d agoAI research

40

Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models

Researchers introduce KoNA, a benchmark exposing vision-language models' failures at selective non-compliance, plus fine-tuning that improves refusal and abstention accuracy.

KoNA is a benchmark for evaluating selective non-compliance in vision-language models across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility and Safety. It tests both query-level and component-level non-compliance using paired single and compound queries, and evaluations across diverse VLMs show models often fail to refuse, correct or abstain appropriately, with failures worsening on compound queries. Fine-tuning VLMs on KoNA examples substantially improves non-compliance accuracy while largely maintaining performance on fully answerable tasks.

Hugging Face daily papers·Sep 3, 20:00 UTC · 12d agoAI research1

45

HyQuant: Hybrid-Precision Quantization for LLM Attention

HyQuant keeps most LLM attention states low-bit while preserving vertical-line tokens and local windows in high precision, maintaining near-lossless accuracy.

HyQuant is a hybrid-precision quantization framework for LLM attention that quantizes most attention states to low bits while keeping accuracy-critical vertical-line tokens and local-window states in full precision, selected via lightweight attention-pattern signals. In the prefill stage it uses a hybrid-precision attention operator, and in the decode stage it applies the same principle to KV-cache compression with fused dequantization and attention computation. Across diverse tasks, models, and datasets it maintains nearly lossless accuracy; code is available on GitHub.

Hugging Face daily papers·Aug 27, 20:00 UTC · 19d agoAI tools & infra1

33

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

A controlled autoregressive testbed shows validation losses must be analyzed per task, and image tokenizer choice affects joint multimodal text modeling.

Researchers built a pure-autoregressive testbed to study image tokenizers as the 'visual language' of unified multimodal models, tracking task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. They found that losses exhibit distinct scaling behavior per task and rank tokenizers differently, and that I2T loss over a shared text vocabulary gives a more consistent loss–performance signal than T2I loss. Better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and tokenizer choice can affect text modeling under joint optimization. Case studies examine the discriminator, semantic supervision, and vocabulary size design axes.

Hugging Face daily papers·Sep 7, 20:00 UTC · 8d agoAI research1

40

MindTopo: Can Foundation Models Reason in Topological Space?

MindTopo benchmark with 11,030 topological tasks shows 14 multimodal LLMs reason better than they plan and remain far below human performance.

MindTopo is a benchmark of topological intuition across five properties grounded in cognitive science and formal topology: continuity, separation, order, enclosure, and knots, evaluated at reasoning and closed-loop planning levels. It contains 11,030 instances across 13 procedurally generated task types with controllable difficulty, benchmarking 14 multimodal LLMs plus agent configurations using image and video generation, including three video generative models. Every MLLM performs better on reasoning than on planning, and the best-performing model remains far below observed human performance. On Qwen3-VL-2B-Instruct, supervised fine-tuning and reinforcement learning improve reasoning more than planning, and audited generated rollouts often fail to follow environment dynamics or preserve topology across transitions.

arXiv cs.AI / cs.LG / cs.CL·Sep 10, 17:54 UTC · 6d agoAI research

33

Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

Latent Interface Training improves robot foundation model generalization by constraining visual conditioning, boosting LIBERO-Plus success up to 10.7 points.

The paper identifies vision-action shortcuts where robot policies exploit task-irrelevant visual cues that fail under distribution shift. Latent Interface Training (LIT) first trains an action expert conditioned on language, robot state, and terminal SE(3) end-effector poses without images, then constrains visual input through a pose-supervised latent interface. Across four VLA and world-action architectures (Pi0.5, MolmoAct2, FAST-WAM, ImageWAM), LIT improves LIBERO-Plus success by 3.87-10.70 percentage points. Real-world tests show 13.30-16.70 percentage-point gains under unseen cameras, lighting, and distractors.

Hugging Face daily papers·Sep 10, 20:00 UTC · 5d agoAI research

32

Context Engineering Inside the Harness: 4 Mechanisms That Beat Context Overflow and Goal Loss on Long-Horizon Tasks

Survey of four harness mechanisms—context budgeting, compaction, todo-state, and memory—that keep long-horizon LLM agents on task across 200+ tool calls.

The article details how agent harnesses, not larger context windows, solve context overflow and goal loss on long-horizon tasks, citing Chroma's Context Rot report showing 18 LLMs (GPT-4.1, Claude 4, Gemini 2.5, Qwen3) degrade on long inputs. Concrete implementations include LangChain Deep Agents offloading tool responses over 20,000 tokens to the filesystem and truncating old tool calls at 85% window usage, and Claude Code capping auto memory at 25KB while re-reading the 5 most recently modified files after compaction. OpenAI's Responses API now offers server-side compaction via context_management with a standalone /responses/compact endpoint, which Codex uses for long-running coding tasks. Manus reports a roughly 100:1 input-to-output token ratio per ~50-tool-call task, motivating todo.md state recitation to prevent goal drift.

MarkTechPost·Sep 13, 05:56 UTC · 3d agoAI research1

52

ConvMem: Convolutional Memory for Long-Context Reasoning

Researchers propose ConvMem, a training-free framework treating LLMs as convolutional kernels for parallelizable long-context reasoning beyond fixed context windows.

ConvMem reformulates long-context reasoning as a hierarchical convolution in which the LLM summarizes text segments hierarchically, shortening the reasoning path from a linear chain to a logarithmic tree. It uses configurable strides, skip connections, and multi-kernel convolution to capture evidence, decompose queries, and enable massive parallelization across segments and reasoning threads. On RULER-HotpotQA and RULER-2WikiMultiHopQA it outperforms training-free baselines and avoids the out-of-distribution overfitting seen in RL-trained approaches like MemAgent.

arXiv cs.AI / cs.LG / cs.CL·Sep 9, 16:53 UTC · 7d agoAI research

38

OracleZoom: On-Policy Self-Distillation Inspired Reference-Constrained Recursive Image Super Resolution

OracleZoom enables recursive extreme-scale image super-resolution via reference-constrained on-policy distillation, reducing hallucinations at deep zoom scales.

OracleZoom tackles recursive super-resolution, where repeated feeding of predictions back into the same model leaves deeper-scale outputs unsupervised as required source resolution grows geometrically. The framework trains on its own trajectory while carrying the last ground-truth evidence beyond the supervision boundary, combining direct and cross-scale supervision, a no-reference quality objective, a KL-constrained pretrained latent prior, and EMA consistency. Across seven datasets it achieves state-of-the-art SR quality across zoom scales, averaging 0.713 CLIPIQA with larger gains at deeper scales and significantly reduced hallucinations. Code, data, and models are publicly released.

Hugging Face daily papers·Sep 5, 20:00 UTC · 10d agoAI research

20