PC-Dr.DPO improves reliability of majority-vote pseudo-labels in test-time RL
Original title: On the Reliability of Majority-Vote Pseudo-Labels in Test-Time Reinforcement Learning
Prompt-Conditional Distributionally Robust DPO makes updates from sampling-sensitive prompts more conservative.
Majority-vote supervision for TTRL decomposes error into population-majority bias and prompt-dependent finite-rollout variation. PC-Dr.DPO conditions within-prompt robust reweighting on observed vote reliability. A cache-based mechanism reduces self-supervised forgetting during online adaptation. Matches or outperforms strongest baseline in 24 of 30 settings.
- Mitigates finite-rollout variation with robust reweighting
- Reduces self-supervised forgetting via cache mechanism
- Tested across five model backbones
- Improves performance on six reasoning benchmarks
From the original article
1 min read at arxiv.orgTest-time reinforcement learning (TTRL) adapts pretrained language models to unlabeled target-domain data using self-generated pseudo-labels, whose reliability remains poorly understood. We analyze majority-vote supervision for TTRL with online Direct Preference Optimization (DPO) and decompose its error into population-majority bias and prompt-dependent finite-rollout variation.
By Haoyu Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.