Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search
Authors:
- 2 March 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 803-816
Abstract
Increasingly complex and diverse deep neural network (DNN) models necessitate distributing the execution across multiple devices for training and inference tasks, and also require carefully planned schedules for performance. However, existing practices often rely on predefined schedules that may not fully exploit the benefits of emerging diverse model-aware operator placement strategies. Handcrafting high-efficiency schedules can be challenging due to the large and varying schedule space. This paper presents Tessel, an automated system that searches for efficient schedules for distributed DNN training and inference for diverse operator placement strategies. To reduce search costs, Tessel leverages the insight that the most efficient schedules often exhibit repetitive pattern (repetend) across different data inputs. This leads to a two-phase approach: repetend construction and schedule completion. By exploring schedules for various operator placement strategies, Tessel significantly improves both training and inference performance. Experiments with representative DNN models demonstrate that Tessel achieves up to 5.5 x training performance speedup and up to 38 % inference latency reduction.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- FLAVA: A Foundational Language And Vision Alignment Model2022
- Swin Transformer V2: Scaling Up Capacity and Resolution2022
- GEMS: GPU-Enabled Memory-Aware Model-Parallelism System for Distributed DNN Training2020
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- Improving performance of nested loops on reconfigurable array processorsACM Transactions on Architecture and Code Optimization, 2012
- Z3: An Efficient SMT Solver2008
- Exact Algorithms for NP-Hard Problems: A Survey2003
- Profile-assisted instruction schedulingInternational Journal of Parallel Programming, 1994
- Register allocation for software pipelined loops1992
- URPR—An extension of URCR for software pipelining1986