| title | Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models |
|---|---|
| authors | Seo Hyun Kim, Sunwoo Hong, Younwoo Choi, Chen-Hao Chao, Se-Young Yun, Rahul G. Krishnan |
| date | 2026-10-02 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.03665 |
| upvotes | 37 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Masked diffusion language models (dLMs) offer a promising parallel alternative to autoregressive models for complex reasoning. However, they face a distinct credit-assignment challenge, since a few commitments during denoising sharply reduce the uncertainty over the remaining masked positions and shape much of the response. Most post-training recipes for dLMs do not use this signal to decide which tokens to train on: they typically train on the final text or assign rewards to whole denoising steps, rather than selecting the individual commitments that shape the response. We introduce Pivot-SD, an efficient offline self-distillation framework that supervises only these high-impact commitments (pivots). Pivot-SD selects pivots using an information-gain metric measuring uncertainty reduction over the remaining masked positions. Pivots from successful trajectories are trained with cross-entropy, and pivots from failed trajectories with targeted unlikelihood, leaving the rest of the failed trajectory untouched. Using only 200 questions and four rollouts each, Pivot-SD improves LLaDA-8B-Instruct over full-sequence SFT and budget-matched diffusion RL baselines across math and code benchmarks. |
cite
@misc{kim2026pivot,
title={Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models},
author={Seo Hyun Kim and Sunwoo Hong and Younwoo Choi and Chen-Hao Chao and Se-Young Yun and Rahul G. Krishnan},
year={2026},
eprint={2610.03665},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.03665}
}