Learning to Solve Hard Problems in RL for LLMs by Never Giving Up
Paper introduces Never Give Up adaptive sampling, fixing RL's 'Matthew Effect' where compute is wasted on easy problems and hard problems see little improvement.
Researchers identify a 'Matthew Effect' in reinforcement learning for LLMs, where RL yields large gains on easy problems but minimal improvement on hard ones because compute is misallocated. They propose Never Give Up (NGU), an adaptive sampling method that keeps generating samples for a problem until one is correct, using asynchronous RL to filter easy problems cheaply and concentrate compute on hard ones. NGU improves performance per compute on the Deepscaler math benchmark and iteratively solves the Manufactoria coding task where standard GRPO with per-test reward fails.
- RL training disproportionately improves easy problems, dubbed the Matthew Effect
- NGU samples until correct, reallocating compute from easy to hard problems
- Uses asynchronous RL and off-policy robustness best practices
- Fully solves Manufactoria coding tasks that GRPO cannot complete
From the original article
1 min read at huggingface.co
We demonstrate that training LLMs with RL does not improve performance equally across a dataset. RL shows large improvements on easy problems that an LLM is already good at solving, but small improvements on hard problems. We call this the Matthew Effect in RL for LLMs, after the phenomenon of cumulative advantage from economics and network science summarized as "the rich get richer".
By Michael Noukhovitch, Hamish Ivison, Nathan Lambert · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.