On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training
Paper shows constraining SFT parameter updates to on-policy update directions (OPSFT) transfers on-policy generalization gains to supervised fine-tuning.
The study finds supervised fine-tuning (SFT) updates parameters along consistent directions while on-policy training continuously adjusts direction during training. The proposed OPSFT method constrains SFT updates to the cumulative direction identified by on-policy paradigms, transferring their generalization advantage to SFT. This combines on-policy generalization with SFT's training efficiency and ability to exploit high-quality trajectories, with only a few on-policy steps needed to identify effective directions.
- SFT updates follow consistent parameter directions; on-policy training adjusts them continuously
- OPSFT constrains SFT to on-policy directions, transferring generalization
- Few on-policy steps suffice to identify generalizing update directions
- Organizations
- OPSFT
From the original article
1 min read at huggingface.co
The strong generalization performance of on-policy post-training paradigms has motivated studies of their parameter update behaviors. However, these studies treat the observed behaviors only as byproducts in on-policy training, overlooking their potential to serve as optimization principles for improving the generalization of other paradigms such as supervised fine-tuning (SFT).
By Shufan Shen, Zhongni Hou, Junshu Sun · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.