Go With The Flow: Churn-Tolerant Decentralized Training of Large Language Models
Preprint
Abstract
Motivated by the emergence of large language models (LLMs) and the importance of democratizing their training, we propose GWTF, the first crash tolerant practical decentralized training framework for LLMs. Differently from existing distributed and federated training frameworks, GWTF enables the efficient collaborative training of a LLM on heterogeneous clients that volunteer their resources. In addition, GWTF addresses node churn, i.e., clients joining or leaving the system at any time, and network instabilities, i.e., network links becoming unstable or unreliable. The core of GWTF is a novel decentralized flow algorithm that finds the most effective routing that maximizes the number of microbatches trained with the lowest possible delay. We extensively evaluate GWTF on GPT-like and LLaMa-like models and compare it against the prior art. Our results indicate that GWTF reduces the training time by up to 45% in realistic and challenging scenarios that involve heterogeneous client nodes distributed over 10 different geographic locations with a high node churn rate.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints2023
- Epidemic Learning: Boosting Decentralized Learning with Randomized Communication2023
- Decentralized Training of Foundation Models in Heterogeneous Environments2022
- Communication-efficient Decentralized Machine Learning over Heterogeneous Networks2021
- DeepChain: Auditable and Privacy-Preserving Deep Learning with Blockchain-based IncentiveIEEE Transactions on Dependable and Secure Computing, 2019
- Screen Savers of the World Unite!Science, 2000
- Optimization by Simulated Annealing: An Experimental Evaluation; Part I, Graph PartitioningOperations Research, 1989
- The Byzantine Generals ProblemACM Transactions on Programming Languages and Systems, 1982
- An Out-of-Kilter Method for Minimal-Cost Flow ProblemsJournal of the Society for Industrial and Applied Mathematics, 1961