SNLP: Layer-Parallel Inference via Structured Newton Corrections

Ligong Han; Kai Xu; Hao Wang; Akash Srivastava

title	SNLP: Layer-Parallel Inference via Structured Newton Corrections
authors	Ligong Han, Kai Xu, Hao Wang, Akash Srivastava
date	2026-05-18
venue	arXiv (preprint, not peer reviewed)
id	arXiv:2605.17842
affiliation	Red Hat AI
upvotes	3 (Hugging Face Daily Papers, 2026-11-01)
links	abs pdf hf code
abstract	Autoregressive language models execute Transformer layers sequentially, creating a latency bottleneck that is not removed by conventional tensor or pipeline parallelism. We study whether this layerwise dependency can be relaxed by treating the hidden-state trace across layers as the solution of a nonlinear residual equation and solving it with parallel Newton-style updates. While this view is principled, exact Newton corrections require expensive Jacobian-vector products and naive fixed-point iterations are unstable on trained Transformers. We introduce Structured Newton Layer Parallelism (SNLP), a training and inference framework that replaces exact layer Jacobians with cheap architecture-induced surrogate dynamics. In residual Transformers, this yields Identity Newton (IDN), where the correction reduces to a prefix-sum-like update; in mHC-style architectures, HC Newton (HCN) uses the model's residual mixing matrix. We further introduce SNLP-aware regularization, which trains models to make one or a few structured Newton iterations accurately approximate the sequential forward. Experiments on nanochat-scale Transformers show that SNLP regularization improves layer-parallel compatibility and can also improve standard sequential perplexity, reducing baseline PPL by 4.7%-23.4%. At inference time, SNLP combined with layer fusion and chunkwise decomposition achieves practical wall-clock speedups: on a 0.5B Nanochat model, it reaches 2.3x speedup while still improving PPL by 6.1%. These results suggest that layer-parallel inference is not merely a numerical approximation to sequential execution, but can act as a useful solver-induced inference bias. We also characterize limitations: off-the-shelf pretrained models are less amenable to this procedure, and exact convergence recovers the sequential computation rather than providing monotonic inference-time scaling.

cite

@misc{han2026snlp,
  title={SNLP: Layer-Parallel Inference via Structured Newton Corrections},
  author={Ligong Han and Kai Xu and Hao Wang and Akash Srivastava},
  year={2026},
  eprint={2605.17842},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2605.17842}
}