Minimally Invasive Steering of Language Models
MISVO steers frozen language models toward a test-time reward while limiting distribution shift via local KL geometry.
Researchers propose Minimally Invasive Steering Vector Optimization (MISVO), which adds vectors to a frozen language model's final hidden states and penalizes interventions using the local KL geometry of the induced token distribution. An analytic Fisher gradient is computed through the frozen model head, and a frozen-reference surrogate keeps parameters unchanged. On preference and code-generation tasks with models of about 1B to 14B parameters, MISVO achieved the highest mean reward in six of seven settings while diversity and coherence stayed near Best-of-N.
- MISVO penalizes steering using the Fisher geometry of the token distribution.
- Model parameters stay frozen; only position-specific steering vectors are optimized.
- Highest mean reward in six of seven model-task settings.
- Diversity and coherence remain close to Best-of-N sampling.
From the original article
1 min read at arxiv.orgPre-logit steering adapts a frozen language model to a test-time reward by adding vectors to its final hidden states. Unregularized reward optimization can substantially alter the output distribution and degrade generation quality. We propose Minimally Invasive Steering Vector Optimization (MISVO), which penalizes interventions using the local KL geometry of the induced token distribution.
By Taha Entesari · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.