Communication-Efficient Distributed Learning: An Overview
Top Cited Papers
- 6 February 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal on Selected Areas in Communications
- Vol. 41 (4) , 851-873
Abstract
Distributed learning is envisioned as the bedrock of next-generation intelligent networks, where intelligent agents, such as mobile devices, robots, and sensors, exchange information with each other or a parameter server to train machine learning models collaboratively without uploading raw data to a central entity for centralized processing. By utilizing the computation/communication capability of individual agents, the distributed learning paradigm can mitigate the burden at central processors and help preserve data privacy of users. Despite its promising applications, a downside of distributed learning is its need for iterative information exchange over wireless channels, which may lead to high communication overhead unaffordable in many practical systems with limited radio resources such as energy and bandwidth. To overcome this communication bottleneck, there is an urgent need for the development of communication-efficient distributed learning algorithms capable of reducing the communication cost and achieving satisfactory learning/optimization performance simultaneously. In this paper, we present a comprehensive survey of prevailing methodologies for communication-efficient distributed learning, including reduction of the number of communications, compression and quantization of the exchanged information, radio resource management for efficient learning, and game-theoretic mechanisms incentivizing user participation. We also point out potential directions for future research to further enhance the communication efficiency of distributed learning in various scenarios.Keywords
Funding Information
- U.S. National Science Foundation (CNS-2128448, 2007714, 2139304, 2146838)
- U.S. Army Research Office under MURI (AG285)
- U.S. Army Research Laboratory Cooperative (W911NF-17-2-0196)
- National Natural Science Foundation of China (62203373)
This publication has 120 references indexed in Scilit:
- DLM: Decentralized Linearized Alternating Direction Method of MultipliersIEEE Transactions on Signal Processing, 2015
- Diffusion LMS Over Multitask NetworksIEEE Transactions on Signal Processing, 2015
- EXTRA: An Exact First-Order Algorithm for Decentralized Consensus OptimizationSIAM Journal on Optimization, 2015
- Distributed Optimization Over Time-Varying Directed GraphsIEEE Transactions on Automatic Control, 2014
- Quantized Subgradient Algorithm and Data-Rate Analysis for Distributed OptimizationIEEE Transactions on Control of Network Systems, 2014
- Multitask Diffusion Adaptation Over NetworksIEEE Transactions on Signal Processing, 2014
- A Regularization Approach to Learning Task Relationships in Multitask LearningACM Transactions on Knowledge Discovery From Data, 2014
- Distributed dual averaging method for multi-agent optimization with quantized communicationSystems & Control Letters, 2012
- Online Learning and Online Convex OptimizationFoundations and Trends in Machine Learning, 2012
- Communication complexity of convex optimizationJournal of Complexity, 1987