SimOPD: Relational Knowledge Transfer for On-Policy Distillation
SimOPD distills teacher knowledge through response similarities, gaining up to 3.1 points cross-architecture.
SimOPD is an on-policy distillation method that supervises a student from a teacher's internal representations as similarity distributions over paired student responses. The student matches those distributions in its own feature space, avoiding hidden-state alignment and cross-model projection. Reported experiments improve the overall average by up to 2.6 percentage points in same-architecture settings and 3.1 points in cross-architecture settings versus the strongest baselines.
- SimOPD transfers teacher knowledge via response-similarity distributions.
- Students match those distributions in their own representation space.
- Gains reach 2.6 and 3.1 points over strongest baselines.
- Method supports same- and cross-architecture distillation without projection.
From the original article
1 min read at arxiv.orgA key challenge in on-policy distillation (OPD) is extracting effective supervision from teacher knowledge. Supervision from token probabilities may underutilize internal knowledge, while direct hidden-state matching forces the student to shift to the teacher's feature space and complicates cross-architecture transfer.
By Kai Tang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.