| title | Function-Space Temporal Self-Distillation for Policy Optimization |
|---|---|
| authors | Aryan Das, Arnab Bhattacharya, Vinay Kumar Verma |
| date | 2026-10-14 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2610.19687 |
| affiliation | Indian Institute of Technology, Kanpur |
| upvotes | 38 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher, yet the teacher evolves with the student, creating unstable targets and motivating additional regularization. We introduce Function-Space Policy Optimization (FSPO), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FSPO stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further introduces privileged disagreement correction to explicitly suppress locally overvalued actions, reliability-adaptive grounding that strengthens reference supervision when self-generated evidence repeatedly fails, and quality and coverage-aware temporal memory that retains verified behaviors and improves privileged demonstrations. Across reasoning and tool-use benchmarks, FSPO achieves the highest average accuracy under a 5-hour training budget, improving over the strongest baseline by approximately 4% points on average while reducing peak memory from about 70 GB to 56 GB on Olmo3-7B-Instruct. |
cite
@misc{das2026function,
title={Function-Space Temporal Self-Distillation for Policy Optimization},
author={Aryan Das and Arnab Bhattacharya and Vinay Kumar Verma},
year={2026},
eprint={2610.19687},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2610.19687}
}