| title | DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss |
|---|---|
| authors | Ruixuan Tu, Yuyang Dong |
| date | 2026-10-13 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.17644 |
| affiliation | SB Intuitions |
| upvotes | 3 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Knowledge distillation (KD) frameworks place teacher and student on the same GPUs, so per-GPU memory limits the teachers that fit. DisaggKD separates them across nodes: teacher replicas stream hidden states to the student, a KD loss over row windows and vocabulary shards equals the full-vocabulary loss in exact arithmetic without forming the full-vocabulary logits, and a prefetch ring paces producer and trainer. Closed-form memory and traffic models locate the memory limits of each layout. On the Qwen3.5-35B-A3B to 9B pair, the disaggregated layout matches the LM loss of the colocated run with the same loss and is faster than that run; our fused loss also lets a colocated run train on as few nodes at faster speed, while the split is memory efficient when the student spans several nodes. On the larger pairs, colocated runs with a fused loss still run OOM where the split trains, because teacher weights and student states fill the budget. |
cite
@misc{tu2026disaggkd,
title={DisaggKD: Efficient Knowledge Distillation through Teacher-Student Disaggregation and a Fused Loss},
author={Ruixuan Tu and Yuyang Dong},
year={2026},
eprint={2610.17644},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.17644}
}