AMPeD: An Analytical Model for Performance in Distributed Training of Transformers
Authors:
- 1 April 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 306-315
Abstract
Transformers are a class of machine learning models that have piqued high interest recently due to a multitude of reasons. They can process multiple modalities efficiently and have excellent scalability. Despite these obvious advantages, training these large models is very time-consuming. Hence, there have been efforts to speed up the training process using efficient distributed implementations. Many different types of parallelism have been identified that can be employed standalone or in combination. However, naively combining different parallelization schemes can incur significant communication overheads, thereby potentially defeating the purpose of distributed training. Thus, it becomes vital to predict the right mapping of different parallelisms to the underlying system architecture. In this work, we propose AMPeD, an analytical model for performance in distributed training of transformers. It exposes all the transformer model parameters, potential parallelism choices (along with their mapping onto the system), the accelerator as well as system architecture specffications as tunable knobs, thereby enabling hardware-software co-design. With the help of 3 case studies, we show that the combinations of parallelisms predicted to be efficient by AMPeD conform with the results from the state-of-the-art literature. Using AMPeD, we also show that future distributed systems consisting of optical communication substrates can train large models up to 4× faster as compared to the current state-of the-art systems without modifying the peak computational power of the accelerators. Finally, we validate AMPeD with in-house experiments on real systems and via published literature. The max. observed error is limited to 12%. The model is available here: https://github.com/CSA-infra/AMPeDKeywords
This publication has 19 references indexed in Scilit:
- Performance prediction for convolutional neural networks on edge GPUs2021
- Silicon circuits for chip‐to‐chip communications in multi‐socket server board interconnectsIET Optoelectronics, 2021
- Transformer based Deep Intelligent Contextual Embedding for Twitter sentiment analysisFuture Generation Computer Systems, 2020
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- Conformer: Convolution-augmented Transformer for Speech Recognition2020
- ASTRA-SIM: Enabling SW/HW Co-Design Exploration for Distributed DL Training Platforms2020
- DeLTA: GPU Performance Model for Deep Learning Applications with In-Depth Memory System Traffic Analysis2019
- Learning Deep Transformer Models for Machine Translation2019
- Performance Modeling and Scalability Optimization of Distributed Deep Learning Systems2015
- Real-Time Performance Prediction for Cloud Components2012