MEPipe: Democratizing LLM Training with Memory-Efficient Slice-Level Pipeline Scheduling on Cost-Effective Accelerators
Open Access
- 30 March 2025
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1263-1278
Abstract
No abstract availableFunding Information
- National Natural Science Foundation of China (U20B2044)
This publication has 16 references indexed in Scilit:
- MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism2023
- Chimera2021
- ZeRO-infinity2021
- Efficient large-scale language model training on GPU clusters using megatron-LM2021
- vPipe: A Virtualized Acceleration System for Achieving Efficient and Scalable Pipeline Parallel DNN TrainingIEEE Transactions on Parallel and Distributed Systems, 2021
- DAPPLE2021
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- Prague: High-Performance Heterogeneity-Aware Asynchronous Decentralized Training2020
- Superneurons2018
- Scaling Distributed Machine Learning with the Parameter Server2014