Beyond Repeated Sampling: Learning Search Policies for LLM Reasoning
An RL-trained concept generator lifts a frozen LLM's pass@k on hard math versus repeated sampling.
The paper argues that repeated independent sampling wastes test-time compute on near-duplicate solutions. It instead samples problem-specific concepts, hints, or strategies and conditions answer generation on them. A small concept generator trained with reinforcement learning raises a larger frozen model's pass@k on hard math at the same answer budget, beats concepts from much larger untuned models, and transfers to generators it was not trained with, including another model family.
- Concept sampling explores semantically instead of local decoding noise.
- Reinforcement learning tunes a small generator for a frozen answer model.
- Pass@k improves versus repeated sampling at the same generation budget.
- The search policy transfers to a different model family.
From the original article
1 min read at arxiv.orgLarge language models increasingly tackle hard reasoning problems by spending more test-time compute, yet the dominant strategy remains naive repeated sampling: draw many independent solutions and hope one is correct. Because such sampling explores only through local decoding noise, it tends to produce many near duplicate attempts rather than genuinely different ideas.
By Ismail Labiad · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.