titleEmoHarness: Boosting Multimodal Emotion Understanding with Meta-Harness Evolving
authorsFenghai Li, Bingxuan Li, Jiawen Zhang, Yifan Shen, Jiateng Liu, Yutong Zhang, Yue Guo
date2026-11-09
venuearXiv (preprint, not peer reviewed)
idarXiv:2611.11994
affiliationUniversity of Illinois at Urbana-Champaign
upvotes2 (Hugging Face Daily Papers, 2026-11-12)
linksabs pdf hf code
cite
relatedHeterogeneous Scientific Foundation Model Collaboration
AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems
Scaling Automatic Research Agents via World Models
Hierarchical Continuous Diffusion Language Models
abstractSmall multimodal models can misjudge emotion when words, voice, and visual cues conflict. We present EmoHarness, which runs such a model, the frozen student, inside a harness: the prompts, settings, and verification code around its calls. A human-written starting harness, the scaffold, numbers the student's visual observations, transcript, and prosody descriptions as citable evidence and has the student weigh candidate emotions against it. In Meta-Harness Evolving (MHE), a stronger LLM, the Builder, reads the student's errors and execution traces offline and edits the harness; static checks and validation gates decide which edits are kept, and the Builder never sees evaluation items. On 5,507 MME-Emotion items, EmoHarness raises task-macro Recognition by 7.6-13.4 points over a single video-only call for all four students and both Builders. Transcripts and prosody descriptions supply most of this boost (+5.38 to +10.71); the scaffold changes the audio-augmented call by -0.79 to +2.54, partly through extra multi-label predictions, and evolved edits add at most +0.95, although no evolved increment survives Holm correction. The harness's emotion context also transfers to IntentQA, with small gains for three of four students, none of which survives correction. EmoHarness thus boosts small multimodal students chiefly by supplying and organizing audio-derived evidence.

bibtex

@misc{li2026emoharness,
  title={EmoHarness: Boosting Multimodal Emotion Understanding with Meta-Harness Evolving},
  author={Fenghai Li and Bingxuan Li and Jiawen Zhang and Yifan Shen and Jiateng Liu and Yutong Zhang and Yue Guo},
  year={2026},
  eprint={2611.11994},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2611.11994}
}