Hierarchical Broadcast Coding: Expediting Distributed Learning at the Wireless Edge
- 7 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Wireless Communications
- Vol. 20 (4) , 2266-2281
Abstract
Distributed learning plays a key role in reducing the training time of modern deep neural networks with massive datasets. In this paper, we consider a distributed learning problem where gradient computation is carried out over a number of computing devices at the wireless edge. We propose hierarchical broadcast coding, a provable coding-theoretic framework to speed up distributed learning at the wireless edge. Our contributions are threefold. First, motivated by the hierarchical nature of real-world edge computing systems, we propose a layered code which mitigates the effects of not only packet losses at the wireless computing nodes but also straggling access points (APs or small base stations). Second, by strategically allocating data partitions to nodes in the overlapping areas between cells, our technique achieves the fundamental lower bound on computational load to combat stragglers. Finally, we take advantage of the broadcast nature of wireless networks by which wireless devices in the overlapping cell coverage broadcast to more than one AP. This further reduces the overall training time in the presence of straggling APs. Experimental results on Amazon EC2 confirm the advantage of the proposed methods in speeding up learning. Our design targets any gradient descent based learning algorithms, including linear/logistic regressions and deep learning.Keywords
Funding Information
- National Research Foundation of Korea (2019R1I1A2A02061135)
This publication has 29 references indexed in Scilit:
- A Survey on Mobile Edge Computing: The Communication PerspectiveIEEE Communications Surveys & Tutorials, 2017
- Speeding Up Distributed Machine Learning Using CodesIEEE Transactions on Information Theory, 2017
- A Scalable Framework for Wireless Distributed ComputingIEEE/ACM Transactions on Networking, 2017
- Mobile Edge Computing: A Survey on Architecture and Computation OffloadingIEEE Communications Surveys & Tutorials, 2017
- Edge-Facilitated Wireless Distributed Computing2016
- A hierarchical edge cloud architecture for mobile computing2016
- Network densification: the dominant theme for wireless evolution into 5GIEEE Communications Magazine, 2014
- Wireless distributed computing: a survey of research challengesIEEE Communications Magazine, 2012
- Femtocell networks: a surveyIEEE Communications Magazine, 2008
- Fundamentals of Wireless Communication2005