Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions
Authors:
- Junbeom KimJKJunbeom Kim
- Kyuyoung KimKKKyuyoung Kim
- Jinwoo ShinJSJinwoo Shin
Preprint
Abstract
Language models trained on web-scale corpora risk memorizing and exposing sensitive information, prompting the need for effective machine unlearning. Prior methods mainly focus on input queries to suppress sensitive outputs, yet this often fails to eliminate the underlying knowledge and limits scalability. To address this, we propose Corrective Unlearning with Retrieved Exclusions (CURE), a novel unlearning framework that verifies model outputs for leakage and revises them into safe responses. Specifically, CURE employs a lightweight corrector that is applied to the original model to verify whether outputs contain target knowledge and to rewrite them if any leakage is detected. To efficiently handle large-scale unlearning requests, CURE retrieves unlearning targets that are relevant to the initial response and provides them as in-context references to the corrector for detection and conditional revision. By leveraging this retrieval augmentation, the corrector can adapt to new unlearning requests without additional training. Extensive evaluations demonstrate that CURE substantially reduces information leakage, even from indirect queries where prior works fall short, while maintaining response quality and general utility. Moreover, it demonstrates robustness under continual unlearning scenarios, making it practical for real-world applications.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Rethinking machine unlearning for large language modelsNature Machine Intelligence, 2025
- RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models2024
- Large Language Model Unlearning via Embedding-Corrupted Prompts2024
- Machine Unlearning of Pre-trained Large Language Models2024
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model2023
- Knowledge Unlearning for Mitigating Privacy Risks in Language Models2023
- Towards Making Systems Forget with Machine Unlearning2015
- The Probabilistic Relevance Framework: BM25 and BeyondFoundations and Trends® in Information Retrieval, 2009
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989