Disentangling Representation Evolution in Transformers through Directional Decomposition

Researchers decompose transformer representation updates into parallel and perpendicular components relative to the hidden state, finding that value-space parallel edits are the most robust for targeted interventions and that suppressing parallel components…

A newly published paper studies transformer representation evolution as functional geometry, decomposing learned updates into parallel and perpendicular components relative to the hidden state, across attention/MLP and value-aggregation spaces. Across pretrained models, the authors find substantial parallel components beyond the residual identity path. Targeted edits reveal a space-dependent asymmetry: exclude-self value-space parallel manipulation is markedly more robust than residual-space and perpendicular counterparts, making it the preferred target for model interventions. As a diagnostic, perpendicular error separates compression methods more clearly than parallel error. When applied as a training intervention, full-aggregate parallel suppression during from-scratch pretraining lowers validation-loss trajectories and improves downstream performance averages, with the value-space variant strongest. Code is released on GitHub. The two source reports (Hugging Face daily papers, 2026-09-13; arXiv cs.AI/cs.LG/cs.CL, 2026-09-14) are consistent, with the arXiv report adding the finding of substantial parallel components beyond the residual identity path across pretrained models.

• Paper decomposes learned transformer updates into parallel and perpendicular components relative to the hidden state, spanning attention/MLP and value-aggregation spaces
• Pretrained models exhibit substantial parallel components beyond the residual identity path
• Exclude-self value-space parallel manipulation is markedly more robust for targeted edits than residual-space and perpendicular alternatives
• Perpendicular error separates compression methods more clearly than parallel error
• Full-aggregate parallel suppression during from-scratch pretraining lowers validation-loss trajectories and improves downstream averages; the value-space variant is strongest
• Code is released on GitHub
• Sources: Hugging Face daily papers (2026-09-13) and arXiv cs.AI/cs.LG/cs.CL (2026-09-14); reports agree, no discrepancies