Activation inversion metamodels scale interpretation of residual stream directions
Original title: Towards interpreting any residual stream direction by scaling activation inversion
MAEMs generate text exemplars that activate supplied residual stream directions better than corpus search or autoencoders.
The paper develops Maximally-Activating Example Metamodels (MAEMs) to interpret any residual stream direction by mapping it to activating text exemplars. The label-free RL objective maximises cosine similarity between the direction and activations of generated text. The Qwen3.6-27B MAEM beats corpus search on held-out languages and domains while activating 89% of held-out SAE features.
- MAEMs outperform corpus search at 10M tokens on majority of held-out languages/domains
- Activates 89% of held-out SAE features
- Verifies concepts via forward pass on subject model
- Recovers LoRA backdoor payloads more often than corpus search
From the original article
1 min read at arxiv.orgTo interpret a direction in a language model's residual stream, we can look for text that strongly activates it. But when such text is found by scanning a corpus, it is limited to what that corpus contains. We develop Maximally-Activating Example Metamodels (MAEMs), language models which map residual stream directions from a subject model to text exemplars that strongly activate those directions.
By Celeste De Schamphelaere · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.