titleLoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers
authorsSiqi Zhu, Suozhi Huang, Yi Zhao, Jiaxuan You
date2026-11-14
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.19649
affiliationUniversity of Illinois at Urbana-Champaign
upvotes4 (Hugging Face Daily Papers, 2026-11-19)
linksabs pdf hf
cite
relatedHeterogeneous Scientific Foundation Model Collaboration
AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
Scaling Automatic Research Agents via World Models
Hierarchical Continuous Diffusion Language Models
abstractLooped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. During autoregressive decoding, however, generating each token requires several sequential passes through these layers, leading to high decoding latency. Pipelined self-speculative methods use shallow-loop predictions to process tokens at different loops in the same batch, but each token must reach the drafting depth before the next can enter the pipeline. We introduce LoopFill, a training-free speculative decoding algorithm for looped LLM that combines multi-token suffix drafts with shallow-loop predictions in a single draft tree. Both drafters contribute to one tree, so a suffix lookup can immediately extend a shallow prediction and several future tokens can enter the pipeline together. The shared block computes the highest-scoring draft candidates at different loop depths in one batch, and full-depth verification preserves the model's autoregressive output distribution. Across seven benchmarks, LoopFill's average speedup over greedy autoregressive decoding is 5.85times on Ouro-2.6B and 5.13times on Huginn-3.5B. The speedups exceed those of LoopAhead, the strongest baseline, by \(40\%\) on Ouro-2.6B and \(25\%\) on Huginn-3.5B.

bibtex

@misc{zhu2026loopfill,
  title={LoopFill: Pipelined Multi-Token Speculative Decoding for Looped Transformers},
  author={Siqi Zhu and Suozhi Huang and Yi Zhao and Jiaxuan You},
  year={2026},
  eprint={2611.19649},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.19649}
}