titleLearning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement
authorsWenbo Huang, Jianxin Bi, Shunyu Yang, Ruiyang Si, Rui Ni, Qiang Wang, Shulong Jiang, Yintianrun Zhang, Feixiang Ruan, Kewei Zhang, Ye Huang, Shukai Gong, Daquan Zhou
date2026-11-05
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.08318
affiliationDAGroup-PKU
upvotes18 (Hugging Face Daily Papers, 2026-11-15)
linksabs pdf hf
cite
relatedRethinking World-Action Model for Compositional and In-Context Robotic Manipulation
PhysLoop: Harnessing LLMs' Physics Awareness for Self-Improving Video Generation Models
PyRUA-DataFarm: LLM Agent as a Cost-Efficient Teleoperator for Real-Ready Robot Data
Fewer Tokens, Better Action: GPT-6 Astra Robot Agents with 14% Higher Success Rate but 65% Fewer Tokens
DC-SAE: Deep Compression Semantic Autoencoder for Faster Diffusion Convergence
abstractSelf-improving embodied agents are increasingly being developed and compared on robot simulation benchmarks. Several recent works expose the benchmark's success criteria to the learning agent, a practice we call Criteria Exposure (CE). These criteria, originally written to verify policies trained by imitation learning, often demand more than the corresponding language instructions, so a score from such a verifier may not faithfully reflect whether the instructed task was completed, especially for robot-control programs written by an agent. As embodied coding agents report ever higher benchmark scores, a natural question arises: when a coding agent is exposed to the benchmark's criteria, does it generate code that better performs the instructed task, or code that simply overfits to the verification criteria? We conduct controlled experiments with a self-improving agent with and without CE, and find that with CE the agent quickly produces code with a higher success rate under the given criteria, but this advantage disappears when the criteria are changed slightly without departing from the original instructions. More importantly, the skills the agent develops under CE transfer poorly across tasks. We further propose instruction-faithful criteria and show that they better reflect the performance of agent-generated programs.

bibtex

@misc{huang2026learning,
  title={Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement},
  author={Wenbo Huang and Jianxin Bi and Shunyu Yang and Ruiyang Si and Rui Ni and Qiang Wang and Shulong Jiang and Yintianrun Zhang and Feixiang Ruan and Kewei Zhang and Ye Huang and Shukai Gong and Daquan Zhou},
  year={2026},
  eprint={2611.08318},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.08318}
}