vTrain: A Simulation Framework for Evaluating Cost-Effective and Compute-Optimal Large Language Model Training
Authors:
- 2 November 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 153-167
Abstract
As large language models (LLMs) become widespread in various application domains, a critical challenge the AI community is facing is how to train these large AI models in a cost-effective manner. Existing LLM training plans typically employ a heuristic based parallel training strategy which is based on empirical observations rather than grounded upon a thorough examination of the search space of LLM parallelization. Such limitation renders existing systems to leave significant performance left on the table, wasting millions of dollars worth of training cost. This paper presents our profiling-driven simulator called vTrain, providing AI practitioners a fast yet accurate software framework to determine an efficient and cost-effective LLM training system configuration. We demonstrate vTrain's practicality through several case studies, e.g., effectively evaluating optimal training parallelization strategies that balances training time and its associated training cost, efficient multi-tenant GPU cluster schedulers targeting multiple LLM training jobs, and determining a compute-optimal LLM model architecture given a fixed compute budget.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- Tale of Two Cs: Computation vs. Communication Scaling for Future Transformers on Future Hardware2023
- ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scale2023
- AMPeD: An Analytical Model for Performance in Distributed Training of Transformers2023
- Lazy Batching: An SLA-aware Batching System for Cloud Machine Learning Inference2021
- ASTRA-SIM: Enabling SW/HW Co-Design Exploration for Distributed DL Training Platforms2020
- SeqPoint: Identifying Representative Iterations of Sequence-Based Neural Networks2020
- FlexReduce: Flexible All-reduce for Distributed Deep Learning on Asymmetric Network Topology2020
- PREMA: A Predictive Multi-Task Scheduling Algorithm For Preemptible Neural Processing Units2020
- GARNET: A detailed on-chip network model inside a full-system simulator2009
- Bandwidth optimal all-reduce algorithms for clusters of workstationsJournal of Parallel and Distributed Computing, 2009