Mystic: Predictive Scheduling for GPU Based Cloud Servers Using Machine Learning
Authors:
- Yash UkidaveYUYash Ukidave
- Xiangyu LiXLXiangyu Li
- David KaeliDavid Kaeli
- 1 May 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 3 (15302075) , 353-362
Abstract
GPUs have become the primary choice of accelerators for high-end data centers and cloud servers, which can host thousands of disparate applications. With the growing demands for GPUs on clusters, there arises a need for efficient co-execution of applications on the same accelerator device. However, the resource contention among co-executing applications causes interference which leads to degradation in execution performance, impacts QoS requirements of applications and lowers overall system throughput. While previous work has proposed techniques for detecting interference, the existing solutions are either developed for CPU clusters, or use static profiling approaches which can be computationally intensive and do not scale well. We present Mystic, an interference-aware scheduler for efficient co-execution of applications on GPU-based clusters and cloud servers. The most important feature of Mystic is the use of learning-based analytical models for detecting interference between applications. We leverage a collaborative filtering framework to characterize an incoming application with respect to the interference it may cause when co-executing with other applications while sharing GPU resources. Mystic identifies the similarities between new applications and the executing applications, and guides the scheduler to minimize the interference and improve system throughput. We train the learning model with 42 CUDA applications, and consider another separate set of 55 diverse, real-world GPU applications for evaluation. Mystic is evaluated on a live GPU cluster with 32 NVIDIA GPUs. Our framework achieves performance guarantees for 90.3% of the evaluated applications. When compared with state-of-the art interference-oblivious schedulers, Mystic improves the system throughput by 27.5% on average, and achieves a 16.3% improvement on average in GPU utilization.Keywords
This publication has 23 references indexed in Scilit:
- NUPAR2015
- Scheduling Multi-tenant Cloud Workloads on Accelerator-Based Systems2014
- Runtime Support for Adaptive Spatial Partitioning and Inter-Kernel Communication on GPUs2014
- Enabling preemptive multiprogramming on GPUs2014
- Paragon2013
- Interference-driven resource management for GPU-based heterogeneous clusters2012
- vCUDA: GPU-Accelerated High-Performance Computing in Virtual MachinesIEEE Transactions on Computers, 2011
- rCUDA: Reducing the number of GPU-based accelerators in high performance clusters2010
- Dense linear algebra solvers for multicore with GPU accelerators2010
- The Scalable Heterogeneous Computing (SHOC) benchmark suite2010