| title | DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models |
|---|---|
| authors | Shuai Wang, Yin Yu, Shengyao Zhuang, Bevan Koopman, Guido Zuccon |
| date | 2026-05-08 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2605.07210 |
| affiliation | The Information Engineering Lab |
| upvotes | 3 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | PromptReps showed that an autoregressive language model can be used directly as a retriever by prompting it to generate dense and sparse representations of a query or passage. Extending this to multiple representatives is inefficient for autoregressive models, since tokens must be generated sequentially, and prior multi-token variants did not reliably improve over single-token decoding. We show that the bottleneck is sequential generation, not the multi-token idea itself. DiffRetriever is a representative-token retriever for diffusion language models: it appends K masked positions to the prompt and reads all K in a single bidirectional forward pass. Across in-domain and out-of-domain evaluation, multi-token DiffRetriever substantially improves over single-token on every diffusion backbone we test, while autoregressive multi-token is flat or negative and pays a latency cost that scales with K where diffusion does not. After supervised fine-tuning, DiffRetriever on Dream is the strongest BEIR-7 retriever in our comparison, ahead of PromptReps, the encoder-style DiffEmbed baseline on the same diffusion backbones, and the contrastively fine-tuned single-vector RepLLaMA. A per-query oracle on the frozen base model exceeds contrastive fine-tuning at the same fixed budget, pointing to adaptive budget selection as future work. Code is available at https://github.com/ielab/diffretriever. |
cite
@misc{wang2026diffretriever,
title={DiffRetriever: Parallel Representative Tokens for Retrieval with Diffusion Language Models},
author={Shuai Wang and Yin Yu and Shengyao Zhuang and Bevan Koopman and Guido Zuccon},
year={2026},
eprint={2605.07210},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2605.07210}
}