| title | Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency |
|---|---|
| authors | Itay Elam, Eliron Rahimi, Avi Mendelson, Chaim Baskin |
| date | 2026-06-05 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.07881 |
| affiliation | Technion Israel institute of technology |
| upvotes | 6 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Pipeline parallelism is essential for training large neural networks, but existing schedules trade off throughput, memory, and optimization consistency. Synchronous pipelines preserve forward/backward weight consistency but suffer from bubbles; asynchronous pipelines remove bubbles but introduce weight-version mismatch, typically requiring weight stashing, prediction, or correction mechanisms. We introduce PACI (Pipeline Asynchronous training with Controlled Inconsistency), a bubble-free asynchronous pipeline method that bounds forward/backward version drift without weight stashing, prediction, additional parameter copies, or global synchronization. The key idea is to use local gradient accumulation as a version-control mechanism: by slowing parameter-version evolution relative to pipeline delay, PACI limits the number of optimizer updates crossed by any micro-batch while preserving steady-state utilization. In GPT-style language-model pretraining, PACI matches the stability and final perplexity of synchronous 1F1B-flush, retains the same peak memory footprint, achieves fully utilized pipeline throughput, and improves training time-to-accuracy by up to 1.69times over the fastest flush baseline. These results show that forward/backward inconsistency need not be eliminated: when explicitly bounded, it can be safely traded for substantial efficiency gains. |
cite
@misc{elam2026breaking,
title={Breaking the Bubble: Asynchronous Pipeline Parallel Training with Bounded Weight Inconsistency},
author={Itay Elam and Eliron Rahimi and Avi Mendelson and Chaim Baskin},
year={2026},
eprint={2606.07881},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.07881}
}