Learning to Steer, Steering to See: Unveiling the Geometry of RLVR in Large Language Models via Trainable Vectors
Vector steering finds a low-dimensional manifold behind RLVR gains and a stabilizer that preserves them.
The paper studies reinforcement learning with verifiable rewards by using vector steering to locate a low-dimensional activation manifold associated with RL-induced reasoning gains. Effective control directions lie mainly in the low-variance complement of the principal subspace, and the geometry stays consistent within a task and base model. Experiments cover five LLMs and six verifiable-reward tasks, with cross-task alignment correlating with capability transfer. Alpha-Stabler monitors principal-subspace intrusion and removes that component from activation gradients, stabilizing training for 2,000 steps and improving RL gains.
- RLVR reasoning gains lie on a small activation manifold
- Control directions sit outside the high-variance principal subspace
- Geometry is consistent across five LLMs and six tasks
- Alpha-Stabler stabilizes training for 2,000 steps and improves gains
- Products
- Alpha-Stabler
From the original article
1 min read at huggingface.co
Reinforcement learning (RL) has become a key paradigm for enhancing the reasoning of large language models, yet the high dimensionality of parameter updates makes its training dynamics hard to analyze. We study reinforcement learning with verifiable rewards (RLVR) and use vector steering to identify a low-dimensional effective manifold in activation space associated with RL-induced gains.
By Yuchen Cai, Ding Cao, Qixiang Yin · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.