titleReframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning
authorsHaozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei
date2026-11-09
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.12886
affiliationNanjing University
upvotes39 (Hugging Face Daily Papers, 2026-11-12)
linksabs pdf hf code
cite
relatedOneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction
AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling
OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools
EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory
Omni-Woodpecker: Hallucination Correction via Evidence-Prior Rebalancing for Omni LLMs
abstractAgentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.

bibtex

@misc{yuan2026reframe,
  title={Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning},
  author={Haozhi Yuan and Handong Li and Yifan Zhang and Qianshan Wei},
  year={2026},
  eprint={2611.12886},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.12886}
}