Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
Researchers propose ERPO, enabling test-time reinforcement learning for code generation via probe-executed consensus rewards, rank masking, and entropy regularization.
The paper introduces probe-driven test-time reinforcement learning (TTRL) for code generation, where output-free probe inputs are constructed from problem statements and candidate programs are executed on them to compute a Probe Consensus Reward (PCR). Because PCR can be gamed through spurious consensus, the authors propose Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which turns low-PCR outcomes into conservative negative updates via rank masking and constrains policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.
- Probe-driven TTRL derives rewards by executing candidate programs on constructed probe inputs, no labeled answers needed.
- Probe Consensus Reward provides behavioral signal for open-vocabulary programs but remains vulnerable to reward hacking.
- ERPO masks low-PCR ranks into conservative negative updates and caps policy drift with an entropy ceiling.
- Method improves pass@1 and pass@k on coding benchmarks in in-domain adaptation and zero-shot transfer.
From the original article
1 min read at arxiv.orgExisting methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal.
By Jiacheng Xu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.