FedOComp: Two-Timescale Online Gradient Compression for Over-the-Air Federated Learning
- 6 April 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 9 (19) , 19330-19345
Abstract
Federated learning (FL) is a machine learning framework, where multiple distributed edge Internet of Things (IoT) devices collaboratively train a model under the orchestration of a central server while keeping the training data distributed on the IoT devices. FL can mitigate the privacy risks and costs from data collection in traditional centralized machine learning. However, the deployment of standard FL is hindered by the expense of the communication of the gradients from the devices to the server. Hence, many gradient compression methods have been proposed to reduce the communication cost. However, the existing methods ignore the structural correlations of the gradients and, therefore, lead to a large compression loss which will decelerate the training convergence. Moreover, many of the existing compression schemes do not enable over-the-air aggregation and, hence, require huge communication resources. In this work, we propose a gradient compression scheme, named FedOComp, which leverages the correlations of the stochastic gradients in FL systems for efficient compression of the high-dimension gradients with over-the-air aggregation. The proposed design can achieve a smaller deceleration of the training convergence compared to other gradient compression methods since the compression kernel exploits the structural correlations of the gradients. It also directly enables over-the-air aggregation to save communication resources. The derived convergence analysis and simulation results further illustrate that under the same power cost, the proposed scheme has a much faster convergence rate and higher test accuracy compared to existing baselines.Keywords
Funding Information
- Huawei Technologies
This publication has 27 references indexed in Scilit:
- Broadband Analog Aggregation for Low-Latency Federated Edge LearningIEEE Transactions on Wireless Communications, 2019
- Federated Machine LearningACM Transactions on Intelligent Systems and Technology, 2019
- Correlation Coefficients: Appropriate Use and InterpretationAnesthesia & Analgesia, 2018
- Modulation-Free M2M Communications for Mission-Critical ApplicationsIEEE Transactions on Signal and Information Processing over Networks, 2017
- Deep Residual Learning for Image Recognition2016
- Principal component analysis: a review and recent developmentsPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2016
- Low-rank approximations for large stationary covariance matrices, as used in the Bayesian and generalized-least-squares analysis of pulsar-timing dataMonthly Notices of the Royal Astronomical Society, 2014
- Online Learning and Online Convex OptimizationFoundations and Trends in Machine Learning, 2012
- Projection-like Retractions on Matrix ManifoldsSIAM Journal on Optimization, 2012
- Message-passing algorithms for compressed sensingProceedings of the National Academy of Sciences of the United States of America, 2009