| title | Federation of Experts: Communication Efficient Distributed Inference for Large Language Models |
|---|---|
| authors | Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini, Philip Levis |
| date | 2026-05-07 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.06206 |
| affiliation | Stanford University |
| upvotes | 1 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Mixture of experts has emerged as the primary mechanism for making Large Language Models (LLMs) computationally efficient. However, in distributed settings, communicating token embeddings between experts is a significant bottleneck. We present the novel Federation of Experts (FoE) architecture. FoE restructures the MoE block of a transformer layer into multiple MoE clusters. Each cluster is responsible for only one of the KV heads and expert parallelism is applied between those experts. Between clusters, a sum synchronizes the post-attention residuals, which then drives routing and dispatch for the next MoE block. In a single-node setting, FoE completely eliminates all-to-all communication as all experts within a group are contained on the same GPU. In multi-node settings, FoE confines all-to-all communication to the intra-node fabric, thus significantly reducing communication overhead. An implementation of FoE finds that on LongBench, FoE significantly improves inference throughput and latency in both single-node and multi-node settings, reducing the end-to-end forward-pass latency by up to 5.2x, TTFT by 3.62x, and TBT by 1.95x. It does so while achieving comparable generation quality to a mixture of experts model of the same size and training configuration. |
cite
@misc{abdurrahman2026federation,
title={Federation of Experts: Communication Efficient Distributed Inference for Large Language Models},
author={Muhammad Shahir Abdurrahman and Chun Deng and Azalia Mirhoseini and Philip Levis},
year={2026},
eprint={2605.06206},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.06206}
}