titlePrompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning
authorsSanket Badhe, Deep Shah
date2026-06-02
venuearXiv (preprint, not peer reviewed)
idarXiv:2602.21103
affiliationGoogle
upvotes8 (Hugging Face Daily Papers, 2026-11-01)
linksabs pdf hf
abstractAdvanced reasoning typically requires Chain-of-Thought prompting, which is accurate but incurs prohibitive latency and substantial test-time inference costs. The standard alternative, fine-tuning smaller models, often sacrifices interpretability while introducing significant resource and operational overhead. To address these limitations, we introduce Prompt-Level Distillation (PLD). We extract explicit reasoning patterns from a Teacher model and organize them into a structured list of expressive instructions for the Student model's System Prompt. Evaluated using Gemma-3 4B, PLD improved Macro F1 scores on StereoSet (57\% to 90.0\%) and Contract-NLI (67\% to 83\%), while increasing LogiQA accuracy to 70\%. Similar results on Mistral Small 3.1 demonstrate cross-architecture generalizability, enabling these compact models to match frontier performance with negligible latency overhead. These expressive instructions render the decision-making process transparent, allowing for full human verification of logic, making this approach ideal for regulated industries such as law, finance, and content moderation, as well as high-volume use cases and edge devices.

cite

@misc{badhe2026prompt,
  title={Prompt-Level Distillation: A Non-Parametric Alternative to Model Fine-Tuning for Efficient Reasoning},
  author={Sanket Badhe and Deep Shah},
  year={2026},
  eprint={2602.21103},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2602.21103}
}