AutoScaffold builds training-time hints for language-model reinforcement learning
Original title: AutoScaffold: AI-Guided Scaffolding for Reinforcement Learning of Language Models
Scaffold-free students beat RL baselines on four tasks, including +7.2 ALFWorld success and +4.0 QA exact match.
AutoScaffold uses a frozen teacher to turn a student's reinforcement-learning experience into adaptive scaffolds, with no human intervention during training. The teacher writes shared strategies and per-task hints, then removes them at evaluation. The same framework was applied to ALFWorld, search-based question answering, mathematical reasoning and GPU kernel generation. Students outperformed the matching RL baselines in all four domains. On math, the student surpassed QuestA's reported accuracy with about one-third as many student training steps.
Unconfirmed: The reported gains, including the QuestA comparison, come from the authors.
- ALFWorld success rose 7.2 points and QA exact match rose 4.0 points.
- Faster correct GPU kernels than the PyTorch reference rose by up to 5.0 points.
- The teacher shifts from workflows to targeted fixes as the student improves.
- Revised scaffolds helped more than the previous scaffold or no scaffold.
From the original article
1 min read at arxiv.orgReinforcement learning with verifiable rewards (RLVR) produces rich training experience, yet outcome rewards reduce each attempt to a single number and leave most of this information unused.
By Zijian Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.