| title | LLMs Get Lost in Evolving User Intent |
|---|---|
| authors | Jihoon Tack, Philippe Laban, Jennifer Neville |
| date | 2026-07-22 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2607.20734 |
| affiliation | Microsoft Research |
| upvotes | 22 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | As LLMs become more capable, they are increasingly deployed as collaborative agents, taking on user-delegated tasks through iterative interaction. Yet genuine interaction is inherently dynamic: users rarely specify their intent upfront, instead disclosing, revising, and reshaping it as the conversation unfolds. Despite this, LLMs are still predominantly evaluated or trained in single-turn, fully-specified settings, leaving open a fundamental question: how well do LLMs track and act on user intent as it evolves over the course of a conversation? To study this, we introduce a framework that transforms static, single-turn tasks into dynamic multi-turn conversations in which the user's intent evolves across turns--incrementally revealed, revised, and at times redirected mid-conversation--while preserving each task's original evaluation protocol, enabling existing benchmarks to be reused as controlled testbeds without new annotation. Across multiple tasks, we surface a consistent phenomenon: strong static-setting performance does not transfer to the evolving-intent setting, with substantial drops across model families. Our findings point to a fundamental gap: today's LLMs do not yet faithfully track and act on the user's evolving intent, a capability invisible to static evaluation yet critical for future collaborative agents. |
cite
@misc{tack2026llms,
title={LLMs Get Lost in Evolving User Intent},
author={Jihoon Tack and Philippe Laban and Jennifer Neville},
year={2026},
eprint={2607.20734},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2607.20734}
}