HuatuoGPT-3: RL-Only Domain Adaptation from Base Models
OnePO does RL-only domain adaptation and produces HuatuoGPT-3, whose 27B model leads reported HealthBench scores.
The paper proposes One-stage Policy Optimization (OnePO) for RL-only domain adaptation, using teacher outputs as temporary guidance rather than a permanent training target. Adaptive Objective Evolution emphasizes informative low-probability teacher tokens, and Teacher Retirement discards teacher outputs once the policy can surpass them. With 20K samples, OnePO scores 67.2 on HealthBench Total, beating SFT+RL by 2.7 points and pure RL by 7.4. Scaled open-source HuatuoGPT-3 reaches 70.1 Total and 71.4 Professional at 27B, which the authors say surpasses GPT-6 Astra.
- OnePO targets gradient starvation and teacher-distribution anchoring
- Scores 67.2 on HealthBench Total with 20K samples
- HuatuoGPT-3 27B reaches 70.1 Total and 71.4 Professional
- Authors say it surpasses GPT-6 Astra; weights and code are released
- Products
- HealthBench
- Organizations
- FreedomIntelligence
- AI models
- HuatuoGPT-3GPT-6 Astra
From the original article
1 min read at huggingface.co
Domain adaptation aims to turn a general-purpose large language model (LLM) into an expert for a target domain. While the dominant SFT+RL pipeline offers a convenient cold start, it may reduce exploration diversity and introduces additional complexity through multi-stage optimization. These limitations motivate RL-only adaptation.
By Junying Chen, Xinyuan Xie, Ziniu Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.