Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing
FOVEATED shifts RoPE context during editing so models recall individual facts from unstructured passages.
Stories per day · 30 days
2 RSS
5 stories, newest first · one card per story
Stories tagged #rope
QuantMLA: Function-Aligned Dual-Path Quantization for Low-Bit MLA KV Caching
QuantMLA jointly quantizes MLA content and RoPE KV caches to INT4 with little accuracy loss and large throughput gains.
Why Do Video Diffusion Models Violate Physics? Unveiling the Flaws in Attention Mechanisms
Researchers trace unphysical video-diffusion motion to RoPE attention decay and propose a frequency-scaling fix.
Distance generalization in transformers: why bother with positional encoding?
arXiv study uses synthetic delay-copy tasks to show how RoPE, ALiBi, NoPE and training data diversity affect transformers' distance generalization.
Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction
Grouped Value Attention stores grouped values and reconstructs content keys via a learned linear map, cutting KV-cache size about 45-47% versus GQA.