| title | Context-Aware RL for Agentic and Multimodal LLMs |
|---|---|
| authors | Peiyang Xu, Bangzheng Li, Sijia Liu, Karthik R. Narasimhan, Pramod Viswanath, Prateek Mittal, Xingyu Fu |
| date | 2026-06-15 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2606.17053 |
| affiliation | Princeton University |
| upvotes | 15 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Large language models (LLMs) often fail when answering requires identifying a small but decisive piece of evidence within a long or complex context, such as a single line in a tool trace or a subtle detail in an image. We propose ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon reasoning and multimodal performance through an indirect auxiliary objective. Instead of supervising only the final answer, ContextRL presents the model with a query, an answer, and two highly similar contexts, and rewards it for selecting the context that supports the query--answer pair, thereby encouraging fine-grained grounding. We construct contrastive context data in two domains: for coding agents, trajectories serve as contexts, yielding 1k pairs built via condition filtering; for multimodal reasoning, images serve as contexts, yielding 7K pairs built via generative editing and similarity search. ContextRL achieves average gains of +2.2% over standard GRPO on 5 long-horizon benchmarks, and +1.8% across 12 diverse visual question answering benchmarks. To disentangle the effect of the proposed objective from that of additional data, we compare against data-augmentation baselines that repurpose the same contrastive contexts as standard query--context--answer examples. These baselines provide little to no improvement, showing that the gains arise from the proposed context-selection objective rather than from the contrastive data alone. |
cite
@misc{xu2026context,
title={Context-Aware RL for Agentic and Multimodal LLMs},
author={Peiyang Xu and Bangzheng Li and Sijia Liu and Karthik R. Narasimhan and Pramod Viswanath and Prateek Mittal and Xingyu Fu},
year={2026},
eprint={2606.17053},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2606.17053}
}