titleRethinking in Spikes: Mitigating Hallucinations in MDLMs with Step-Aware Decoding
authorsZhongxing Xu, Zhonghua Wang, Zhe Qian, Shiyan Su, Ming Hu, Xiaocheng Zou, Wei Feng, Mingquan Lin, Yuyin Zhou, Yifan Peng, Hamid Rezatofighi, Zongyuan Ge
date2026-10-14
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.12901
affiliationCornell University
upvotes36 (Hugging Face Daily Papers, 2026-11-18)
linksabs pdf hf code
cite
relatedCorrect Labels Are Not Enough: Reliable Metadata Can Silently Control Model Behavior
SceneAligner: 3D-Grounded Floorplan Localization in the Wild
Inside the Trading Loop: Markets, Models, and Moves with LLM Agents
Raster2Seq: Polygon Sequence Generation for Floorplan Reconstruction
Quantum Boltzmann machine learning of ground-state energies
abstractRecent advancements in multimodal diffusion language models (MDLMs) have exhibited strong global modeling capabilities through iterative refinement. We observe that mask uncertainty follows an overall downward trend during iterative denoising, while a few denoising steps exhibit localized entropy spikes closely associated with hallucinations. We argue that anomalous steps can be identified by measuring the entropy deviation of each denoising step from its neighboring temporal window. We hypothesize that discrete token commitment discards the distributional information at masked positions, limiting the continued interaction between candidate semantics and the context during entropy spikes. With this goal, we present StepRefine, a plug-and-play decoding strategy that leverages semantic context to achieve reliable denoising. The core of our method lies in a step-aware switch between discrete denoising and latent refinement. The model employs probability-weighted embeddings at entropy spike steps to preserve diverse reasoning hypotheses and performs multiple rounds of continuous refinement. Meanwhile, a refinement controller monitors distributional stability to reduce unnecessary loops and switch the model back to standard discrete denoising. Through extensive experiments, StepRefine demonstrates significant hallucination mitigation across different MDLMs on multimodal benchmarks.

bibtex

@misc{xu2026rethinking,
  title={Rethinking in Spikes: Mitigating Hallucinations in MDLMs with Step-Aware Decoding},
  author={Zhongxing Xu and Zhonghua Wang and Zhe Qian and Shiyan Su and Ming Hu and Xiaocheng Zou and Wei Feng and Mingquan Lin and Yuyin Zhou and Yifan Peng and Hamid Rezatofighi and Zongyuan Ge},
  year={2026},
  eprint={2611.12901},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.12901}
}