RLVR models abandon reasoning strategies under OOD shifts
Original title: Not All Reasoning Generalizes: A Case Study on Learning Generalizable Reasoning Strategies in RLVR
Co-training with bijective inversion tasks restores derivation and OOD accuracy.
Reinforcement learning with verifiable rewards enables generalizable improvement in LLM reasoning but fails under easy-to-hard OOD shifts. The model learns in-distribution structured derivations yet collapses to brute-force enumeration, causing sharp accuracy drops. Indirect co-training with bijective tasks mitigates the failure by making derivation more certain and substantially restoring OOD performance.
- Model succeeds in-distribution with correct reasoning strategy
- Bijective tasks are not uniformly harder yet help OOD shifts
- Outcome-only RLVR implicitly credits effective reasoning strategies
From the original article
1 min read at arxiv.orgReinforcement learning with verifiable rewards (RLVR) can enable generalizable improvement for large language model (LLM) reasoning. This paper studies how to incentivize such generalization via reasoning strategies. We study this question with a controllable equation-solving framework: finding multiple inputs x for equations f(x)=y, where f is a non-bijective string function.
By Ziqi Pang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.