DREAM: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads
Open Access
- 25 March 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 73-86
Abstract
Emerging real-time multi-model ML (RTMM) workloads such as AR/VR and drone control involve dynamic behaviors in various granularity; task, model, and layers within a model. Such dynamic behaviors introduce new challenges to the system software in an ML system since the overall system load is not completely predictable, unlike traditional ML workloads. In addition, RTMM workloads require real-time processing, involve highly heterogeneous models, and target resource-constrained devices. Under such circumstances, developing an effective scheduler gains more importance to better utilize underlying hardware considering the unique characteristics of RTMM workloads. Therefore, we propose a new scheduler, DREAM, which effectively handles various dynamicity in RTMM workloads targeting multi-accelerator systems. DREAM quantifies the unique requirements for RTMM workloads and utilizes the quantified scores to drive scheduling decisions, considering the current system load and other inference jobs on different models and input frames. DREAM utilizes tunable parameters that provide fast and effective adaptivity to dynamic workload changes. In our evaluation of five scenarios of RTMM workload, DREAM reduces the overall UXCosT, which is an equivalent metric of the energy-delay product (EDP) for RTMM defined in the paper, by 32.2% and 50.0% in the geometric mean (up to 80.8% and 97.6%) compared to state-of-the-art baselines, which shows the efficacy of our scheduling methodology.Keywords
All Related Versions
This publication has 44 references indexed in Scilit:
- Resource-aware hybrid scheduling algorithm in heterogeneous distributed computingFuture Generation Computer Systems, 2015
- Dynamic Heterogeneity-Aware Resource Provisioning in the CloudIEEE Transactions on Cloud Computing, 2014
- Overload management in real-time control applications using (m, k)-firm guaranteeIEEE Transactions on Parallel and Distributed Systems, 1999
- Combined task and message scheduling in distributed real-time systemsIEEE Transactions on Parallel and Distributed Systems, 1999
- An optimal algorithm for scheduling soft aperiodic tasks in dynamic-priority preemptive systemsIEEE Transactions on Software Engineering, 1997
- Allocation and scheduling of precedence-related periodic tasksIEEE Transactions on Parallel and Distributed Systems, 1995
- New strategies for assigning real-time tasks to multiprocessor systemsIEEE Transactions on Computers, 1995
- The deferrable server algorithm for enhanced aperiodic responsiveness in hard real-time environmentsIEEE Transactions on Computers, 1995
- Scheduling processes with release times, deadlines, precedence and exclusion relationsIEEE Transactions on Software Engineering, 1990
- The Spring kernel: a new paradigm for real-time operating systemsACM SIGOPS Operating Systems Review, 1989