| title | Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach |
|---|---|
| authors | Jiarui Qin, Junjie Huang, Yunjia Xi, Di Yin, Yong Yu, Weinan Zhang, Xing Sun, Weiwen Liu |
| date | 2026-10-14 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.19425 |
| affiliation | Tencent |
| upvotes | 10 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments. We introduce ASAP (Agent Scaffold Adaptation for Pretraining Evaluation), a lightweight framework for eliciting and evaluating agentic behavior from pretrained base models on end-to-end tasks. Rather than converting base models into chat-style agents, ASAP preserves the completion interface and adapts existing agent scaffolds through structured trajectory continuation, short reference prefixes, and minimal recovery mechanisms. Across software-engineering and deep-research benchmarks, ASAP yields discriminative end-to-end performance across base models and pretraining checkpoints. Beyond aggregate scores, ASAP exposes model-generated execution trajectories, enabling analysis of exploration, editing, testing, recovery, and task completion across training stages. These trajectories provide a behavioral view of how agent-relevant capabilities change during pretraining and how execution patterns differ after post-training. Our results suggest that end-to-end agent evaluation can complement conventional pretraining benchmarks by providing a practical diagnostic of interactive model behavior. |
cite
@misc{qin2026evaluating,
title={Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach},
author={Jiarui Qin and Junjie Huang and Yunjia Xi and Di Yin and Yong Yu and Weinan Zhang and Xing Sun and Weiwen Liu},
year={2026},
eprint={2610.19425},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.19425}
}