The information geometry of large language models is shared, learned, and controllable
Fisher-Rao output geometry is shared across LLM architectures and enables minimum-disturbance interventions that transfer across prompts for steering and editing.
The paper shows the Fisher-Rao geometry of next-token probabilities, unlike activation geometry, is consistent across transformer, state-space, and recurrent models and supports semantic-category transfer. Agreement with human word choices rises with accuracy, scale, and training, and corpus statistics predict held-out fact acquisition. The geometry prescribes minimum-disturbance local interventions whose updates transfer from donor prompts to unseen prompts, improving steering, editing, attribution, dictionary learning, and fine-tuning versus Euclidean control.
- Output geometry (Fisher-Rao) is shared across transformer, SSM, and RNN architectures
- Geometry prescribes minimum-disturbance behavioral interventions with predictable cost
- Interventions learned on donor prompts transfer to unseen prompts
- Single geometric correction improves steering, editing, attribution, and fine-tuning
From the original article
1 min read at huggingface.co
Large language models learn similar behaviours, yet it remains unclear what structure they share or how to change one behaviour without disturbing others. The Fisher-Rao geometry of next-token probabilities connects these questions: behaviour determines this geometry up to output-preserving symmetries, whereas activation geometry depends on coordinates.
By Dario Picozzi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.