GIST improves offline RLVR accuracy by selective token supervision
Original title: GIST: Gibbs-Targeted Hierarchical Data Selection for Offline RLVR
GIST approximates Gibbs-optimal policy to select informative responses and tokens, outperforming baselines on GSM8K MATH-500 MBPP while supervising 66.6% fewer tokens.
The paper analyzes token informativeness in offline RLVR using Gibbs-optimal policy from KL-regularized RLVR. Correct responses contribute via forward KL on low-prob tokens while incorrect responses contribute via reverse KL on high-prob tokens. GIST uses sampling temperature and initial policy token probabilities for hierarchical selection, raising average accuracy on three benchmarks for three models under both tested objectives.
- GIST selects correct responses on-policy and incorrect responses near policy mode
- Supervises only mismatch tokens using initial policy probabilities
- Outperforms seven selection baselines on GSM8K MATH-500 MBPP
- 66.6% fewer tokens supervised than full training
- Works with two offline training objectives
From the original article
1 min read at arxiv.orgOffline reinforcement learning with verifiable rewards (RLVR) trains on a fixed pool of sampled responses and typically supervises every token of every response, although responses and tokens are not equally informative.
By Xuliang Huang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.