| title | Demystifying Agent Skills: Why They Work-Until They Don't |
|---|---|
| authors | Zhiyuan Jiang, Fangrui Huang, Hanwen Xing, Xander Wu, Yipeng Gao, Rui Cao, Mengdi Wang, Shilong Liu, Yijiang Li |
| date | 2026-08-14 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.14036 |
| affiliation | University of California at San Diego |
| upvotes | 56 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf code |
| abstract | Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \textbf{When do skills help, why do they work, and where do they fail?} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\% of skill cases, versus 4.5\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\% to 3.3\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents. |
cite
@misc{jiang2026demystifying,
title={Demystifying Agent Skills: Why They Work-Until They Don't},
author={Zhiyuan Jiang and Fangrui Huang and Hanwen Xing and Xander Wu and Yipeng Gao and Rui Cao and Mengdi Wang and Shilong Liu and Yijiang Li},
year={2026},
eprint={2608.14036},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.14036}
}