UVeQFed: Universal Vector Quantization for Federated Learning
Top Cited Papers
- 23 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Signal Processing
- Vol. 69 (1053587X) , 500-514
Abstract
Traditional deep learning models are trained at a centralized server using data samples collected from users. Such data samples often include private information, which the users may not be willing to share. Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their data. FL consists of an iterative procedure, where in each iteration the users train a copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. A major challenge that arises in this method is the need of each user to repeatedly transmit its learned model over the throughput limited uplink channel. In this work, we tackle this challenge using tools from quantization theory. In particular, we identify the unique characteristics associated with conveying trained models over rate-constrained channels, and propose a suitable quantization scheme for such settings, referred to as universal vector quantization for FL (UVeQFed). We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion. We then theoretically analyze the distortion, showing that it vanishes as the number of users grows. We also characterize how models trained with conventional federated averaging combined with UVeQFed converge to the model which minimizes the loss function. Our numerical results demonstrate the gains of UVeQFed over previously proposed methods in terms of both distortion induced in quantization and accuracy of the resulting aggregated model.Keywords
All Related Versions
Funding Information
- Israel Science Foundation (0100101)
- Horizon 2020 Framework Programme (646804-ERC-COG-BNYQ)
- National Natural Science Foundation of China (NSFC-61629101)
This publication has 32 references indexed in Scilit:
- Multiterminal source coding with high resolutionIEEE Transactions on Information Theory, 1999
- QuantizationIEEE Transactions on Information Theory, 1998
- Optimization of lattices for quantizationIEEE Transactions on Information Theory, 1998
- Results on lattice vector quantization with ditheringIEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing, 1996
- On lattice quantization noiseIEEE Transactions on Information Theory, 1996
- A vector quantization approach to universal noiseless coding and quantizationIEEE Transactions on Information Theory, 1996
- On universal quantization by randomized uniform/lattice quantizersIEEE Transactions on Information Theory, 1992
- On universal quantizationIEEE Transactions on Information Theory, 1985
- Voronoi regions of lattices, second moments of polytopes, and quantizationIEEE Transactions on Information Theory, 1982
- The rate-distortion function for source coding with side information at the decoderIEEE Transactions on Information Theory, 1976