Decentralized SGD with Over-the-Air Computation
Authors:
- 1 December 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
We consider multiple devices with local datasets collaboratively learning a global model through device-to-device (D2D) communications. The conventional decentralized stochastic gradient descent (DSGD) solution for this problem assumes error-free orthogonal links among the devices. This is based on the assumption of an underlying communication protocol that takes care of the noise, fading, and interference in the wireless medium. In this work, we show the suboptimality of this approach by designing the communication and learning protocols jointly. We first consider a point-to-point (P2P) communication scheme by scheduling D2D transmissions in an orthogonal fashion to minimize interference. Then, we propose a novel over-the-air consensus scheme by exploiting the signal superposition property of wireless transmission, rather than avoiding interference. In the proposed OAC-MAC scheme, multiple nodes align their transmissions toward a single receiver node. For both schemes, we cast the scheduling problem as a graph coloring problem. We then numerically compare the two approaches for the distributed MNIST image classification task under various network conditions. We show that the OAC-MAC scheme attains better convergence speed and final accuracy thanks to the improved robustness against channel fading and noise. We also introduce a noise-aware version of the OAC-MAC scheme with further improvements in the convergence speed and accuracy.Keywords
Funding Information
- BEACON (677854)
This publication has 10 references indexed in Scilit:
- Communicate to Learn at the EdgeIEEE Communications Magazine, 2020
- Decentralized Federated Learning via SGD over Wireless D2D Networks2020
- On Analog Gradient Descent Learning Over Multiple Access Fading ChannelsIEEE Transactions on Signal Processing, 2020
- Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-AirIEEE Transactions on Signal Processing, 2020
- Federated Learning Over Wireless Fading ChannelsIEEE Transactions on Wireless Communications, 2020
- Federated Learning via Over-the-Air ComputationIEEE Transactions on Wireless Communications, 2020
- Machine Learning in the AirIEEE Journal on Selected Areas in Communications, 2019
- On the Convergence of Decentralized Gradient DescentSIAM Journal on Optimization, 2016
- Dual Averaging for Distributed Optimization: Convergence Analysis and Network ScalingIEEE Transactions on Automatic Control, 2011
- Distributed Subgradient Methods for Multi-Agent OptimizationIEEE Transactions on Automatic Control, 2009