titleSecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation
authorsYibo Peng, Long Lian, David Wagner, Sizhe Chen
date2026-08-21
venuearXiv (preprint, not peer reviewed)
idarXiv:2608.21500
upvotes3 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf code
abstractPrompt injection is listed as the \#1 threat to AI agents. When an agent accesses external data from websites, files, or emails, an attacker may inject a prompt into the data, saying, "Ignore all prior instructions and perform <an attacker's task>." To prevent arbitrary manipulation of agents, defenders try to train secure LLMs, which, however, still suffer from near 100% attack success rates (ASRs) against adaptive prompt injections. We note that this is because existing defensive finetuning recipes rely on sequence-level feedback signals (in DPO or GRPO). Treating an entire output equally prevents the model from learning precisely which output tokens are insecure. In this paper, we propose Secure On-Policy Distillation (SecOPD) that provides token-level feedback to guide defensive fine-tuning. The LLM receives an injected sample and produces a rollout, whose tokens are scored by the initialization model given the corresponding clean input. With more fine-grained training signals, our defended Qwen3.6-27B achieves a 9.0% ASR against the SoTA PISmith adaptive prompt injections, compared to 94.0% for the prior SoTA, Meta-SecAlign. The obtained security generalizes to domains completely unseen in training: in agentic tool calling, SecOPD achieves a 4.7% ASR compared to 5.5% for Meta-SecAlign. Code and the model are available at https://github.com/pppyb/SecOPD and https://huggingface.co/pybbb/Qwen3.6-27B-SecOPD.

cite

@misc{peng2026secopd,
  title={SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillation},
  author={Yibo Peng and Long Lian and David Wagner and Sizhe Chen},
  year={2026},
  eprint={2608.21500},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2608.21500}
}