SpongeTraining: Achieving High Efficiency and Accuracy for Wireless Edge-Assisted Online Distributed Learning
- 25 February 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 22 (8) , 4930-4945
Abstract
Edge-assisted Distributed Learning (EDL) is a popular machine learning paradigm that uses a set of distributed edge nodes to collaboratively train a machine learning model using training data. Most of existing works implicitly assume that the fixed amount of training data is pre-collected and dispatched from user devices to edge nodes. In real world, however, training data in edge nodes are collected from user devices through wireless networks, and the volume and distribution of training data in edge nodes could exhibit temporal and spatial fluctuations due to varying wireless situations (e.g., network congestion, link capacity variation). In this way, existing solutions suffer from slow convergence and low accuracy. In this paper, we propose SpongeTraining to achieve high efficiency and accuracy for online EDL. To accommodate to fluctuations in training data, SpongeTraining uses a buffer at each worker to store received training data and adaptively adjusts training batch size and learning rate of each worker based on training data extracted from the buffer. Experiment results based on real-world datasets show that SpongeTraining outperforms existing solutions by accelerating the training process up to 50% for reaching the same training accuracy.Keywords
Funding Information
- Key-Area Research and Development Program of Guangdong Province (2021B0101400001)
- National Natural Science Foundation of China (62002019, 62002066)
- Beijing Institute of Technology Research Fund Program for Young Scholars
- China Postdoctoral Science Foundation (2021M690705)
- Shanghai Pujiang Program (2020PJD005)
This publication has 27 references indexed in Scilit:
- An Adaptive Batch-Orchestration Algorithm for the Heterogeneous GPU Cluster Environment in Distributed Deep Learning System2018
- Cyclical Learning Rates for Training Neural Networks2017
- Edge Computing: Vision and ChallengesIEEE Internet of Things Journal, 2016
- Deep Residual Learning for Image Recognition2016
- Distributed Online Convex Optimization Over Jointly Connected DigraphsIEEE Transactions on Network Science and Engineering, 2014
- Online distributed optimization via dual averaging2013
- Understanding Characteristics of Available Bandwidth in Wireless EnvironmentProcedia Computer Science, 2011
- An optimal method for stochastic composite optimizationMathematical Programming, 2011
- Primal-dual first-order methods with ${\mathcal {O}(1/\epsilon)}$ iteration-complexity for cone programmingMathematical Programming, 2009
- Convergence Analysis of a Proximal-Like Minimization Algorithm Using Bregman FunctionsSIAM Journal on Optimization, 1993