| title | The More Popular, The Harder to Forget: Adaptive Popularity for LLM Unlearning |
|---|---|
| authors | Anna Borisiuk, Andrey Savchenko, Alexander Panchenko, Elena Tutubalina |
| date | 2026-08-14 |
| venue | arXiv (preprint, not peer reviewed) |
| id | arXiv:2608.14229 |
| upvotes | 17 (Hugging Face Daily Papers, 2026-11-01) |
| links | abs pdf hf |
| abstract | Popular facts are memorised more deeply during pretraining and resist removal longer than rare ones, yet existing LLM unlearning methods apply uniform gradient pressure regardless of training-data frequency. We propose the AdaPop (Adaptive Popularity) method, which combines local token confidence with a per-fact popularity-dependent exponent derived from an external proxy (e.g., Wikidata sitelinks, LLM-as-Judge), and automates the forget-retain balance via a dual-ascent controller that adjusts the retain penalty each epoch. Across three model families and two benchmarks, AdaPop leaks ~5x less forgotten content than competing methods under paraphrased queries and ~1.6x less under adversarial reformulations. We support our analysis with internal metrics: under our method, forget-set hidden states move further from the pre-unlearning model's states than under other methods, while retain-set representations remain close. |
cite
@misc{borisiuk2026more,
title={The More Popular, The Harder to Forget: Adaptive Popularity for LLM Unlearning},
author={Anna Borisiuk and Andrey Savchenko and Alexander Panchenko and Elena Tutubalina},
year={2026},
eprint={2608.14229},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2608.14229}
}