Privacy-Preserving Split Learning with Vision Transformers using Patch-Wise Random and Noisy CutMix
Preprint
Abstract
In computer vision, the vision transformer (ViT) has increasingly superseded the convolutional neural network (CNN) for improved accuracy and robustness. However, ViT's large model sizes and high sample complexity make it difficult to train on resource-constrained edge devices. Split learning (SL) emerges as a viable solution, leveraging server-side resources to train ViTs while utilizing private data from distributed devices. However, SL requires additional information exchange for weight updates between the device and the server, which can be exposed to various attacks on private training data. To mitigate the risk of data breaches in classification tasks, inspired from the CutMix regularization, we propose a novel privacy-preserving SL framework that injects Gaussian noise into smashed data and mixes randomly chosen patches of smashed data across clients, coined DP-CutMixSL. Our analysis demonstrates that DP-CutMixSL is a differentially private (DP) mechanism that strengthens privacy protection against membership inference attacks during forward propagation. Through simulations, we show that DP-CutMixSL improves privacy protection against membership inference attacks, reconstruction attacks, and label inference attacks, while also improving accuracy compared to DP-SL and DP-MixSL.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Mix2SFL: Two-Way Mixup for Scalable, Accurate, and Communication-Efficient Split Federated LearningIEEE Transactions on Big Data, 2023
- Rethinking Architecture Design for Tackling Data Heterogeneity in Federated LearningConference on Computer Vision and Pattern Recognition (CVPR), 2022
- An adaptive federated learning scheme with differential privacy preservingFuture Generation Computer Systems, 2022
- Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of ThingsIEEE Transactions on Computers, 2021
- Rethinking Spatial Dimensions of Vision Transformers2021
- Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and ApplicationsProceedings of the IEEE, 2021
- Hiding in the Crowd: Federated Data Augmentation for On-Device LearningIEEE Intelligent Systems, 2020
- End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things2020
- Federated Learning: Challenges, Methods, and Future DirectionsIEEE Signal Processing Magazine, 2020
- Distributed learning of deep neural network over multiple agentsJournal of Network and Computer Applications, 2018