DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads
Authors:
- Seah KimSKSeah Kim
- Hyoukjun KwonHKHyoukjun Kwon
- Jinook SongJSJinook Song
- Jihyuck JoJJJihyuck Jo
- Yu-Hsin ChenYCYu-Hsin Chen
- Liangzhen LaiLLLiangzhen Lai
- Vikas ChandraVCVikas Chandra
Preprint
Abstract
Emerging real-time multi-model ML (RTMM) workloads such as AR/VR and drone control involve dynamic behaviors in various granularity; task, model, and layers within a model. Such dynamic behaviors introduce new challenges to the system software in an ML system since the overall system load is not completely predictable, unlike traditional ML workloads. In addition, RTMM workloads require real-time processing, involve highly heterogeneous models, and target resource-constrained devices. Under such circumstances, developing an effective scheduler gains more importance to better utilize underlying hardware considering the unique characteristics of RTMM workloads. Therefore, we propose a new scheduler, DREAM, which effectively handles various dynamicity in RTMM workloads targeting multi-accelerator systems. DREAM quantifies the unique requirements for RTMM workloads and utilizes the quantified scores to drive scheduling decisions, considering the current system load and other inference jobs on different models and input frames. DREAM utilizes tunable parameters that provide fast and effective adaptivity to dynamic workload changes. In our evaluation of five scenarios of RTMM workload, DREAM reduces the overall UXCost, which is an equivalent metric of the energy-delay product (EDP) for RTMM defined in the paper, by 32.2% and 50.0% in the geometric mean (up to 80.8% and 97.6%) compared to state-of-the-art baselines, which shows the efficacy of our scheduling methodology.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- End‐to‐end global to local convolutional neural network learning for hand pose recovery in depth dataIET Computer Vision, 2021
- Convolutional Networks with Adaptive Inference Graphs2018
- SkipNet: Learning Dynamic Routing in Convolutional Networks2018
- SSD: Single Shot MultiBox Detector2016
- Resource-aware hybrid scheduling algorithm in heterogeneous distributed computingFuture Generation Computer Systems, 2015
- An optimal algorithm for scheduling soft aperiodic tasks in dynamic-priority preemptive systemsIEEE Transactions on Software Engineering, 1997
- Allocation and scheduling of precedence-related periodic tasksIEEE Transactions on Parallel and Distributed Systems, 1995
- New strategies for assigning real-time tasks to multiprocessor systemsIEEE Transactions on Computers, 1995
- The deferrable server algorithm for enhanced aperiodic responsiveness in hard real-time environmentsIEEE Transactions on Computers, 1995
- Scheduling processes with release times, deadlines, precedence and exclusion relationsIEEE Transactions on Software Engineering, 1990