| title | Dynamic Multi-Byte Prediction With Hierarchical Language Models |
|---|---|
| authors | Abraham Toluwase Owodunni, Chibuzor Okocha, Christan Grant, Tomasz Limisiewicz, Sachin Kumar |
| date | 2026-08-16 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.15454 |
| affiliation | The Ohio State University |
| upvotes | 19 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Byte-level hierarchical language models (LMs) have recently emerged as a robust alternative to their popular counterparts that use subword tokenization. However, generating one byte at a time remains a bottleneck for inference speed. To address this, we introduce multi-byte prediction (MBP), which generates multiple bytes in parallel, speeding up inference with minimal performance impact and no additional parameters. MBP builds on the popular multi-token prediction (MTP) paradigm with two crucial innovations. First, we introduce a variable-length prediction window that aligns with the latent tokens, or segments, of a hierarchical LM. Second, we implement a novel attention-masking scheme that enables parallel byte prediction without violating causality. We show that multi-byte prediction strikes a Pareto-optimal trade-off across multiple generative tasks, instruction following, question answering, summarization, and machine translation, achieving the best trade-off between performance and inference throughput. |
cite
@misc{owodunni2026dynamic,
title={Dynamic Multi-Byte Prediction With Hierarchical Language Models},
author={Abraham Toluwase Owodunni and Chibuzor Okocha and Christan Grant and Tomasz Limisiewicz and Sachin Kumar},
year={2026},
eprint={2608.15454},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.15454}
}