DisaggKD splits teacher and student across nodes for distillation
Original title: DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss
On the Qwen3.5-35B-A3B to 9B pair, the split matches colocated LM loss and is faster.
DisaggKD places teacher replicas and the student on separate nodes so per-GPU memory no longer limits teacher size. Teachers stream hidden states; a windowed, vocabulary-sharded loss equals the full-vocabulary loss in exact arithmetic, and a prefetch ring paces training. On Qwen3.5-35B-A3B distilled to 9B, the split matched the colocated language-model loss and ran faster. On larger pairs, colocated fused-loss runs still ran out of memory where the split trained.
Unconfirmed: Speed and memory claims are the authors' reported runs, without independent confirmation.
- A fused loss also lets a colocated run train on fewer nodes faster.
- The split is memory efficient when the student spans several nodes.
- Closed-form memory and traffic models locate each layout's limits.
- Products
- DisaggKD
- AI models
- Qwen3.5-35B-A3B
From the original article
1 min read at huggingface.co
Knowledge distillation (KD) frameworks place teacher and student on the same GPUs, so per-GPU memory limits the teachers that fit. DisaggKD separates them across nodes: teacher replicas stream hidden states to the student, a KD loss over row windows and vocabulary shards equals the full-vocabulary loss in exact arithmetic without forming the full-vocabulary logits, and a prefetch ring paces producer and trainer.
By Ruixuan Tu, Yuyang Dong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.