Lucid: A Non-intrusive, Scalable and Interpretable Scheduler for Deep Learning Training Jobs
Authors:
Open Access
- 27 January 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 457-472
Abstract
While recent deep learning workload schedulers exhibit excellent performance, it is arduous to deploy them in practice due to some substantial defects, including inflexible intrusive manner, exorbitant integration and maintenance cost, limited scalability, as well as opaque decision processes. Motivated by these issues, we design and implement Lucid, a non-intrusive deep learning workload scheduler based on interpretable models. It consists of three innovative modules. First, a two-dimensional optimized profiler is introduced for efficient job metric collection and timely debugging job feedback. Second, Lucid utilizes an indolent packing strategy to circumvent interference. Third, Lucid orchestrates resources based on estimated job priority values and sharing scores to achieve efficient scheduling. Additionally, Lucid promotes model performance maintenance and system transparent adjustment via a well-designed system optimizer. Our evaluation shows that Lucid reduces the average job completion time by up to 1.3× compared with state-of-the-art preemptive scheduler Tiresias. Furthermore, it provides explicit system interpretations and excellent scalability for practical deployment.Keywords
This publication has 45 references indexed in Scilit:
- Borg, Omega, and KubernetesQueue, 2016
- Network-Aware Scheduling for Data-Parallel Jobs2015
- Deep learningNature, 2015
- Reservation-based Scheduling2014
- Apache Hadoop YARN2013
- Accurate intelligible models with pairwise interactions2013
- Jockey2012
- Dryad2007
- Random ForestsMachine Learning, 2001
- A guided tour to approximate string matchingACM Computing Surveys, 2001