titleNarrowSpec: Draft Hidden Width Need Not Match the Target in Speculative Decoding
authorsAlagappan Valliappan, Ankit More, Dheevatsa Mudigere, Jatin Mitra
date2026-11-18
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.23835
affiliationNVIDIA
upvotes3 (Hugging Face Daily Papers, 2026-11-21)
linksabs pdf hf
cite
relatedCosmos 3: Omnimodal World Models for Physical AI
LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding
LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation
SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning
SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness
abstractSpeculative decoding is widely used to speed up generation in large language models (LLMs), with a small draft model proposing tokens that one forward pass of the target model verifies. Draft models that read the target's hidden states, such as DFlash and EAGLE-3, typically inherit the target's hidden width. We show that the draft's whole residual stream can run at half or a quarter of that width while largely preserving acceptance. For DFlash, which borrows the target's vocabulary projection, the trained output adapter folds into it, giving one narrower projection with logits unchanged up to floating-point rounding. We test two drafter families, DFlash and EAGLE-3, on two targets, Qwen3-8B (NVFP4) and GPT-OSS-20B (MXFP4), in SGLang on a B200 and a DGX Spark (GB10). At batch size 1 and temperature 0, narrowing the draft to half or a quarter width speeds up end-to-end decoding by 1.06-1.12x on B200 and 1.13-1.44x on GB10. The speedup holds at larger batch sizes. From batch 32 to the largest batch each GPU fits, narrowing reaches up to 1.21x on both GPUs, and no narrow draft is slower than full width.

bibtex

@misc{valliappan2026narrowspec,
  title={NarrowSpec: Draft Hidden Width Need Not Match the Target in Speculative Decoding},
  author={Alagappan Valliappan and Ankit More and Dheevatsa Mudigere and Jatin Mitra},
  year={2026},
  eprint={2611.23835},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.23835}
}