Mixed-Privacy Forgetting in Deep Networks
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 792-801
Abstract
We show that the influence of a subset of the training samples can be removed – or "forgotten" – from the weights of a network trained on large-scale image classification tasks, and we provide strong computable bounds on the amount of remaining information after forgetting. Inspired by real-world applications of forgetting techniques, we introduce a novel notion of forgetting in mixed-privacy setting, where we know that a "core" subset of the training samples does not need to be forgotten. While this variation of the problem is conceptually simple, we show that working in this setting significantly improves the accuracy and guarantees of forgetting methods applied to vision classification tasks. Moreover, our method allows efficient removal of all information contained in non-core data by simply setting to zero a subset of the weights with minimal loss in performance. We achieve these results by replacing a standard deep network with a suitable linear approximation. With opportune changes to the network architecture and training procedure, we show that such linear approximation achieves comparable performance to the original network and that the forgetting problem becomes quadratic and can be solved efficiently even for large models. Unlike previous forgetting methods on deep networks, ours can achieve close to the state-of-the-art accuracy on large scale vision tasks. In particular, we show that our method allows forgetting without having to trade off the model accuracy.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks2020
- Formalizing Data Deletion in the Context of the Right to Be Forgotten2020
- Deep Learning with Differential Privacy2016
- Deep Residual Learning for Image Recognition2016
- Towards Making Systems Forget with Machine Unlearning2015
- The Algorithmic Foundations of Differential PrivacyFoundations and Trends® in Theoretical Computer Science, 2014
- Fast Exact Multiplication by the HessianNeural Computation, 1994