TERM jailbreak defense modulates residuals from attention entropy
Compared with seven defenses on seven attacks, it reports stronger robustness and low over-refusal at modest inference cost.
Stories per day · 30 days
4 RSS
Relatedlanguage-models4inference3kv-cache3llm-defense3llm-inference3long-context3transformers3jailbreak2over-refusal2video-diffusion2agent-security1alphafold1arxiv1born-measurements1bottleneck-rank1classification1compression1continuum-attention1
58 stories, newest first · one card per story
Stories tagged #attention
Paper trains Vision Transformers to specialize spatial attention heads
Two annotation-free focusing losses steer multi-head attention toward distinct image regions for denser predictions.
Muir improves linear attention with gating and readout
Muir reduces perplexity by 1.8-2.0% over GDN-2 and supports 10M-token extrapolation.
Systolic Linear Attention optimizes gated delta attention for Google TPUs
The method achieves 1.6x to 2.5x faster performance on TPU v7x and 11-24% faster training for MoE models compared to KDA.
LowRAM compresses KV cache with low-rank keys and amortised values
Test-time value predictors plus cross-layer key compression give about 4× compression and up to 5× decode throughput.
Born-Measured Attention provides exact word evidence in transformers
Born measurements enable exact integrated-gradient evidence for word contributions with 76.98% e-SNLI accuracy.
DepthGraft shares KV hierarchically to regularize deep LLMs
On a 16B MoE model the authors report +7.0 points on MMLU-Pro while cutting KV cache by about one third.
Show HN derives a FlashAttention kernel from about 25 lines of SQL
The Hacker News feed only shows the title, 20 points, and one comment.
Flide accelerates decision inference with exact state forking
Flide decouples logical tree structure from execution in a flattened token stream using ForkAttention, achieving up to 9.5x speedup over vLLM while preserving exact probabilities.
ViT-HarDis conditions attention on hard samples for face recognition
ViT-HarDis applies stronger head decorrelation to ambiguous samples, outperforming uniform regularization on IJB-B and TinyFace.
Hybrid wave-attention beats tiny transformers on perplexity, not memory
A 10.94M hybrid scores 5.95 perplexity, but its peak VRAM matches the pure transformer.
AB-Attention approximates dense video attention at high sparsity
On Wan2.1 and HunyuanVideo the authors report skipping 89–92% of attention with up to 2.23× speedup.
GAAC losslessly compresses BF16 KV caches about 1.41×
On NVIDIA DGX Spark, 64K-token decode attention sped up 1.19–1.27× and resident KV memory fell about 26%.
Heads of Security attention gates improve prompt-injection robustness
Authors report RL-Hammer attack success of 10% on HoS versus 97% on GPT-OSS-20B and 85% on SecAlign.
Paper frames relational attention as lattice gauge theory
Untrained holonomy attention infers unobserved S4 relations at 0.88 with a fifth of edges corrupted.
Stanford and UC Berkeley introduce TokenGate for faster long-context prefill
The title claims token-gated attention speeds long-context prefill by up to 11.3 times; the article body was not available.
1
Paper derives attention architectures from geometric knowledge spaces
The framework recovers standard, causal, linear, and tensor attention plus recurrent networks, then proposes new geometric spaces.
Edge-aware attention cuts graph-generation GPU memory about 65%
A FlexAttention implementation kept generation quality near the baseline while improving training and sampling throughput.
How Finite-Rank Bottlenecks Shape the Low-Rank Adaptation Landscape
A theory paper shows frozen-activation bottleneck rank, not sample size, controls why LoRA updates stay low-rank.
Transformers forget prompts as generated context grows
Hidden-state distance between two prompts decays as a power law; trained models forget faster than untrained ones.
MJev-Spatial: System One Spatial Intelligence via Native Attention Matching
MJev-Spatial classifies spatial answers on Qwen3.5-4B without decoding tokens, topping twelve benchmarks and running much faster.
Velox: Spatial Memory Makes World Models Efficient
Velox speeds long video world-model rollouts 2.43x at chunk 48 while staying near dense quality.
First Passage Attention selectively erases intermediate context
A 110M-parameter model matched FoX perplexity and perfectly retrieved passkeys at eight times training length with a compressed cache.
Continuum Attention
Continuum Attention beats softmax and focal attention by integrating a continuous memory curve.