Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspective
Open Access
- 29 September 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 32 (6) , 1-26
Abstract
Deep learning (DL) has become a key component of modern software. In the “ big model ” era, the rich features of DL-based software (i.e., DL software) substantially rely on powerful DL models, e.g., BERT, GPT-3, and the recently emerging GPT-4, which are trained on the powerful cloud with large datasets. Hence, training effective DL models has become a vital stage in the whole software lifecycle. When training deep learning models, especially those big models, developers need to parallelize and distribute the computation and memory resources amongst multiple devices (e.g., a cluster of GPUs) in the training process, which is known as distributed deep learning training , or distributed training for short. However, the unique challenges that developers encounter in distributed training process have not been studied in the software engineering community. Given the increasingly heavy dependence of current DL-based software on distributed training, this paper aims to fill in the knowledge gap and presents the first comprehensive study on developers’ issues in distributed training. To this end, we focus on popular DL frameworks that support distributed training (including TensorFlow, PyTorch, Keras, and Horovod) and analyze 1,131 real-world developers’ issues about using these frameworks reported on Stack Overflow and GitHub. We construct a fine-grained taxonomy consisting of 30 categories regarding the fault symptoms and summarize common fix patterns for different symptoms. We find that: (1) many distributed-specific faults and non-distributed-specific faults inherently share the same fault symptoms, making it challenging to debug; (2) most of the fault symptoms have frequent fix patterns; (3) about half of the faults are related to system-level configurations. Based on the results, we suggest actionable implications on research avenues that can potentially facilitate the distributed training to develop DL-based software, such as focusing on the frequent and common fix patterns when designing testing or debugging tools, developing efficient testing and debugging techniques for communication configuration along with the synthesis of network configuration analysis, designing new multi-device checkpoint-and-replay techniques to help reproduction, and designing serverless APIs for cloud platforms.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62172008 and 62102009)
- National Natural Science Fund for the Excellent Young Scientists Fund Program
- Center for Data Space Technology and System, Peking University
- ERC Advanced Grant (741278)
This publication has 12 references indexed in Scilit:
- A generic communication scheduler for distributed DNN training acceleration2019
- Lineage stash2019
- A comprehensive study on deep learning bug characteristics2019
- Editor’s NoteScience China Information Sciences, 2016
- Scalable Parallel Programming with CUDAQueue, 2008
- On the Benjamini–Hochberg methodThe Annals of Statistics, 2006
- Qualitative methods in empirical studies of software engineeringIEEE Transactions on Software Engineering, 1999
- Coefficient Kappa: Some Uses, Misuses, and AlternativesEducational and Psychological Measurement, 1981
- The Measurement of Observer Agreement for Categorical DataBiometrics, 1977
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960