E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models

Arseny Ivanov; Alexander Kolesov; Alexander Korotin; Ivan Oseledets; Mikhail Goncharov

title	E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models
authors	Arseny Ivanov, Alexander Kolesov, Alexander Korotin, Ivan Oseledets, Mikhail Goncharov
date	2026-09-29
venue	arXiv (preprint, not peer reviewed)
id	arXiv:2609.37533
upvotes	44 (Hugging Face Daily Papers, 2026-11-01)
links	abs pdf hf code
abstract	Masked diffusion models (MDMs) generate sequences by progressively unmasking several tokens per denoising step, but their reverse process is typically factorized over positions, limiting sample quality in the few-step regime where diffusion's speed advantage over autoregressive decoding matters most. A recent line of work introduces a continuous Gaussian latent, trained as a variational autoencoder, to capture correlations across positions, but such approaches are prone to posterior collapse, where the latent is silently ignored. We propose Enhanced Mixture-of-Experts (E-MoE), which builds the reverse process as a mixture of factorized distributions over a discrete shared latent given by the expert-routing decisions of a Mixture-of-Experts (MoE) backbone, without increasing active parameters over the factorized baseline. Across synthetic multi-modal benchmarks, binarized MNIST, and LM1B, E-MoE improves few-step generation over factorized baselines.

cite

@misc{ivanov2026e,
  title={E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models},
  author={Arseny Ivanov and Alexander Kolesov and Alexander Korotin and Ivan Oseledets and Mikhail Goncharov},
  year={2026},
  eprint={2609.37533},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2609.37533}
}