titleOn the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer
authorsZhenyi Shen, Junru Lu, Wei Liu, Jiangnan Ye, Jiazheng Li, Di Yin, Xing Sun, Tameem Adel, Yali Du, Lin Gui, Yulan He
date2026-11-04
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.06976
affiliationKing's College London
upvotes14 (Hugging Face Daily Papers, 2026-11-08)
linksabs pdf hf code
cite
relatedSelf-Distillation with Verbal Regularization
Large Language Models Hack Rewards, and Society
Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs
EnvLunch: The Environment as a Free Lunch for Learning to Interact
HRDPath: an explainable multi-model deep learning architecture for predicting homologous recombination deficiency from histopathology images
abstractPrefill-to-Autoregressive Cross-model Transfer (PACT), a form of latent communication between language models, combines a larger model's prompt representations with a smaller model's efficient generation, offering a promising way to balance capability and inference cost. However, existing systems provide limited support for independently evaluating large-small model collaboration, and improved task performance alone does not establish whether the receiver benefits from the sender's information. We introduce PLatour, an open-source PACT framework that connects frozen models through lightweight, zero-initialized linear bridges trained directly with task rewards and supports hybrid-attention architectures. PLatour significantly outperforms open-source baselines on generation tasks and surpasses Mostik's reported results on shared mathematics and coding benchmarks using a substantially smaller sender. However, controlled experiments show that the gains on some tasks can be reproduced by uninformative sender signals, self-communication, or receiver-only training. We find that trainable bridges can act as parameter-efficient adaptation mechanisms without exploiting useful sender information. We therefore propose PACT-Check to quantify how much of a trained PACT system's gain genuinely comes from communication. Finally, we discuss the opportunities and challenges of PACT systems. Code is available at https://github.com/zhenyi4/platour.

bibtex

@misc{shen2026opportunities,
  title={On the Opportunities and Challenges of Prefill-to-Autoregressive Cross-Model Transfer},
  author={Zhenyi Shen and Junru Lu and Wei Liu and Jiangnan Ye and Jiazheng Li and Di Yin and Xing Sun and Tameem Adel and Yali Du and Lin Gui and Yulan He},
  year={2026},
  eprint={2611.06976},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.06976}
}