EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

Minseo Kim; Minjae Lee; Seunghyuk Oh; Kevin Galim; Donghoon Kim; Coleman Hooper; Harman Singh; Amir Gholami; Hyung Il Koo; Wonjun Kang

title

authors

Minseo Kim, Minjae Lee, Seunghyuk Oh, Kevin Galim, Donghoon Kim, Coleman Hooper, Harman Singh, Amir Gholami, Hyung Il Koo, Wonjun Kang

date

2026-06-17

venue

arXiv (preprint, not peer reviewed)

arXiv:2606.18967

affiliation

FuriosaAI

upvotes

15 (Hugging Face Daily Papers, 2026-11-01)

links

abs pdf hf code

abstract

Reinforcement learning (RL) has become a representative post-training paradigm for LLMs, enabling strong reasoning and agentic capabilities. However, rollout generation remains a dominant latency bottleneck because autoregressive sampling decodes responses sequentially and a small number of long-tailed generations often determine completion time. Speculative decoding (SD) offers a natural way to address this bottleneck, as it is a well-established technique for serving fixed LLMs that reduces latency by rapidly drafting tokens and accepting them through parallel verification while preserving the target-model distribution. However, its practical speedups do not directly carry over to RL rollouts: (i) the evolving target policy makes any fixed drafter increasingly mismatched with the policy's output distribution; and (ii) active batch sizes shrink throughout rollout decoding, shifting decoding from compute-bound to memory-bound regimes where parallel verification can exploit underutilized compute. Therefore, accelerating RL rollouts requires both a drafter that remains effective under long, high-temperature generations from an evolving policy and system-aware use of SD that avoids compute-bound regimes. We present EfficientRollout, a system-aware self-SD framework designed to address this gap for RL rollouts. EfficientRollout induces a quantized drafter from the target model (i.e. self-speculative decoding), keeping it coupled to the evolving policy without separate drafter pretraining or online adaptation. It further coordinates a system-aware SD toggle policy with acceptance-aware draft-length adaptation, enabling speculation only in beneficial regimes while matching the drafting budget to evolving drafter quality. EfficientRollout reduces rollout and end-to-end latency by up to 19.6% and 12.7%, respectively, over an accelerated AR rollout baseline, while preserving final model quality.

@misc{kim2026efficientrollout,
  title={EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts},
  author={Minseo Kim and Minjae Lee and Seunghyuk Oh and Kevin Galim and Donghoon Kim and Coleman Hooper and Harman Singh and Amir Gholami and Hyung Il Koo and Wonjun Kang},
  year={2026},
  eprint={2606.18967},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2606.18967}
}