Computation vs. Communication Scaling for Future Transformers on Future Hardware
Preprint
Abstract
Scaling DNNs is shown to deliver dramatic quality gains across ML problems. This, however, has also led to a concomitant quadratic increase in computation cost. To tackle this, along with the failure of accelerator memory capacity to keep up, training these models increasingly relies on distributed training techniques. As such, an important question of interest is: how will compute and communication relatively scale as models scale and hardware evolves? A careful study which answers this question can better guide the design of future systems. To this end, this work provides a comprehensive multi-axial (algorithmic, empirical, hardware evolution) analysis of compute vs. communication (Comp-vs.-Comm) scaling for future Transformer models on future hardware. Using algorithmic analysis we show that compute generally enjoys an edge over communication as models scale. However, when viewed through the lens of slower memory capacity scaling, these trends are being stressed. Next, we craft an empirical strategy to study Comp-vs.-Comm scaling for future models/hardware using existing hardware. This allows hundreds of future models/hardware scenarios to be studied at three orders of magnitude lower profiling costs. Our experiments demonstrate that communication will be a significant portion (about 40-75%) of execution as models and hardware evolve, and communication which is today hidden by overlapped computation will likely get exposed. Further, the generality of our strategy makes it a strong basis to perform Comp-vs.-Comm scaling analysis for any future model. Overall, this work underscores the increasingly large role communication will play as models scale.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- A Survey of Quantization Methods for Efficient Neural Network Inference2022
- ZeRO-infinity2021
- Demystifying the MLPerf Training Benchmark Suite2020
- DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference2020
- An In-Network Architecture for Accelerating Shared-Memory Multiprocessor Collectives2020
- MLPerf Inference Benchmark2020
- ERNIE 2.0: A Continual Pre-Training Framework for Language UnderstandingProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Deep Learning Language Modeling Workloads: Where Time Goes on Graphics Processors2019
- Benchmarking and Analyzing Deep Neural Network Training2018
- Scalable Hierarchical Aggregation Protocol (SHArP): A Hardware Architecture for Efficient Data Reduction2016