BACombo—Bandwidth-Aware Decentralized Federated Learning
Open Access
- 5 March 2020
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 9 (3) , 440
Abstract
The emerging concern about data privacy and security has motivated the proposal of federated learning. Federated learning allows computing nodes to only synchronize the locally- trained models instead of their original data in distributed training. Conventional federated learning architecture, inherited from the parameter server design, relies on highly centralized typologies and large nodes-to-server bandwidths. However, in real-world federated learning scenarios, the network capacities between nodes are highly uniformly distributed and smaller than that in data centers. As a result, how to efficiently utilize network capacities between computing nodes is crucial for conventional federated learning. In this paper, we propose Bandwidth Aware Combo (BACombo), a model segment level decentralized federated learning, to tackle this problem. In BACombo, we propose a segmented gossip aggregation mechanism that makes full use of node-to-node bandwidth for speeding up the communication time. Besides, a bandwidth-aware worker selection model further reduces the transmission delay by greedily choosing the bandwidth-sufficient worker. The convergence guarantees are provided for BACombo. The experimental results on various datasets demonstrate that the training time is reduced by up to 18 times that of baselines without accuracy degrade.Keywords
This publication has 11 references indexed in Scilit:
- AWStream2018
- EMNIST: Extending MNIST to handwritten letters2017
- Ako2016
- WANalytics2015
- MALT2015
- Scaling Distributed Machine Learning with the Parameter Server2014
- A peer-to-peer recommender system for self-emerging user communities based on gossip overlaysJournal of Computer and System Sciences, 2013
- Bandwidth optimal all-reduce algorithms for clusters of workstationsJournal of Parallel and Distributed Computing, 2009
- Finite-time Analysis of the Multiarmed Bandit ProblemMachine Learning, 2002
- The Nonstochastic Multiarmed Bandit ProblemSIAM Journal on Computing, 2002