Reconstructing Individual Data Points in Federated Learning Hardened with Differential Privacy and Secure Aggregation
Authors:
- 1 July 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 241-257
Abstract
Federated learning (FL) is a framework for users to jointly train a machine learning model. FL is promoted as a privacy-enhancing technology (PET) that provides data minimization: data never "leaves" personal devices and users share only model updates with a server (e.g., a company) coordinating the distributed training. While prior work showed that in vanilla FL a malicious server can extract users’ private data from the model updates, in this work we take it further and demonstrate that a malicious server can reconstruct user data even in hardened versions of the protocol. More precisely, we propose an attack against FL protected with distributed differential privacy (DDP) and secure aggregation (SA). Our attack method is based on the introduction of sybil devices that deviate from the protocol to expose individual users’ data for reconstruction by the server. The underlying root cause for the vulnerability to our attack is a power imbalance: the server orchestrates the whole protocol and users are given little guarantees about the selection of other users participating in the protocol. Moving forward, we discuss requirements for privacy guarantees in FL. We conclude that users should only participate in the protocol when they trust the server or they apply local primitives such as local DP, shifting power away from the server. Yet, the latter approaches come at significant overhead in terms of performance degradation of the trained model, making them less likely to be deployed in practice.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Practical Secure Aggregation for Privacy-Preserving Machine Learning2017
- Deep Models Under the GAN2017
- Prochlo2017
- Privacy-Preserving Deep Learning: Revisited and Enhanced2017
- Membership Inference Attacks Against Machine Learning Models2017
- Deep Learning with Differential Privacy2016
- Why Google stores billions of lines of code in a single repositoryCommunications of the ACM, 2016
- Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?IEEE Transactions on Signal Processing, 2016
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiersInternational Journal of Security and Networks, 2015