titleEvidence-Based Credit Assignment for Search-Augmented Language Models
authorsWen Wang, Jindong Wang, Jiahua Bao, Tu Yongsiqi, Yihao Liu, Junwei He, Wenkui Fan
date2026-11-18
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.23824
affiliationTencent
upvotes6 (Hugging Face Daily Papers, 2026-11-21)
linksabs pdf hf
cite
relatedEvolvingWorld: An Open-Schema Framework for Co-Evolving Role-Play Agents and World Model in Interactive Literary World
A New Role for Relevance: Guiding Corpus Interaction in Agentic Search
WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report
FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention
VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?
abstractReinforcement learning from terminal answer rewards enables search-augmented language models to interleave reasoning and retrieval without step-level annotations, yet outcome supervision assigns identical credit to every search in a trajectory regardless of whether it retrieved useful information. This limitation is especially severe when all sampled rollouts for a question fail, collapsing group-centered outcome advantages to zero even when trajectories gather markedly different evidence. We present Evidence-based Credit Assignment (ECA), which augments terminal outcome rewards with a training-only auxiliary signal measuring how retrieved documents improve the likelihood of a reference answer. Rather than scoring the full dialogue prefix, ECA queries a frozen snapshot of the current policy under an evidence-only prompt containing solely the question and the deduplicated documents observed so far, isolating the utility of retrieved passages from surface variations in generated thoughts or search queries. Within this framework, we compare two return formulations, ECA-Step (which accumulates step-clipped potential increments) and ECA-Horizon (which preserves the telescoping horizon gain in potential), paired with cross-fold group normalization, novelty gating, and search-local token assignment. Across seven Search QA benchmarks, ECA-Horizon improves strict macro exact match over an outcome-only GRPO baseline by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct, respectively, outperforming both RLOO and GiGPO. Zero-shot evaluation on MultiHop-RAG further demonstrates strong transfer to multi-hop inference queries.

bibtex

@misc{wang2026evidence,
  title={Evidence-Based Credit Assignment for Search-Augmented Language Models},
  author={Wen Wang and Jindong Wang and Jiahua Bao and Tu Yongsiqi and Yihao Liu and Junwei He and Wenkui Fan},
  year={2026},
  eprint={2611.23824},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.23824}
}