Improving Atomic-Fact Recall via Focused Views in Unstructured Knowledge Editing
FOVEATED shifts RoPE context during editing so models recall individual facts from unstructured passages.
Unstructured knowledge editing often lets models reproduce an edited passage while failing to recall its individual facts without that context, a failure called context reliance. The authors argue the passage-level objective underestimates later facts because richer preceding context lowers their initial loss. FOVEATED, applied only during editing, randomly shifts Rotary Position Embedding positions of preceding-context keys to create focused sentence views, then restores native positional encoding at inference. It improves five editors on two LLM backbones and three benchmarks for both direct-optimization and locate-then-edit methods.
- Edited models can recite a passage yet fail to recall its atomic facts.
- Later facts look easier because richer context lowers their initial loss.
- RoPE key positions of prior context are shifted only while editing.
- Improvements hold across five editors, two LLM backbones, and three benchmarks.
From the original article
1 min read at huggingface.co
Large language models (LLMs) increasingly serve as general-purpose interfaces to factual knowledge, but their parameters do not automatically reflect information that changes after pretraining. Knowledge editing (KE) provides a targeted alternative to costly retraining by modifying selected knowledge and preserving unrelated knowledge and general capabilities.
By Ding Wu, Ye Zhang, Haoyu Wang · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.