Language models integrate temporal anchors into factual recall
Original title: Tracing How Language Models Integrate Temporal Anchors into Factual Recall
Temporal anchors resolve context-dependent facts via subject-mediated or direct MLP routes in open-weight models.
Anchored factual recall requires models to select objects valid under explicit conditions like time. Causal interventions identify two parallel routes: subject-mediated and direct attention to anchors. Contributions from both routes add up depending on the query and relation.
- Anchor resolves final subject-token representation.
- Relation redistributes readout across attention heads.
- Upper-layer MLPs turn anchor signal into time-appropriate preference.
- Routes are parallel and balance depends on relation.
- Consistent across four open-weight models.
From the original article
1 min read at arxiv.orgFactual recall in language models is typically framed as retrieving a factual association: a subject--relation pair maps to a single object. However, many facts are context-dependent: the same pair maps to different objects depending on a condition such as time. We introduce anchored factual recall, in which the model receives an explicit conditioning expression, an anchor, and must select the object valid under it.
By Ayana Niwa · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.