ActionOPD: Distilling Decisions, Not Tokens
ActionOPD distills black-box teacher decisions into Qwen3 agents, gaining 3.3-6.9 points out of distribution.
ActionOPD is an on-policy distillation method that supervises language-model agents at the semantic-action level instead of copying teacher tokens. At student-visited states it samples actions from the student and a black-box teacher and rewards semantic agreement, without teacher token probabilities or a shared tokenizer. Training Qwen3-4B and Qwen3-8B with two teachers on curated tool-use tasks yields average gains of 3.3 to 6.9 percentage points on four out-of-distribution benchmarks, and competitive τ²-bench results against offline and token-level distillation.
- ActionOPD aligns semantic actions rather than teacher tokens or reasoning traces.
- It needs neither teacher token probabilities nor a shared tokenizer or model family.
- Qwen3-4B and Qwen3-8B gain 3.3-6.9 points on four out-of-distribution benchmarks.
- On τ²-bench it is competitive with offline distillation and token-level OPD.
From the original article
1 min read at huggingface.co
Distilling capable language-model agents into smaller models requires aligning the student's behavior with the teacher's. This raises a fundamental question: at which states and at what granularity should this alignment be performed? Offline imitation provides dense but token-specific supervision and can suffer from state-distribution mismatch.
By Huining Yuan, Xiangmin Yi, Zelai Xu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.