SDRM3: A Dynamic Scheduler for Dynamic Real-time Multi-model ML Workloads
Authors:
Preprint
Abstract
Emerging real-time multi-model ML (RTMM) workloads such as AR/VR and drone control often involve dynamic behaviors in various levels; task, model, and layers (or, ML operators) within a model. Such dynamic behaviors are new challenges to the system software in an ML system because the overall system load is unpredictable unlike traditional ML workloads. Also, the real-time processing requires to meet deadlines, and multi-model workloads involve highly heterogeneous models. As RTMM workloads often run on resource-constrained devices (e.g., VR headset), developing an effective scheduler is an important research problem. Therefore, we propose a new scheduler, SDRM3, that effectively handles various dynamicity in RTMM style workloads targeting multi-accelerator systems. To make scheduling decisions, SDRM3 quantifies the unique requirements for RTMM workloads and utilizes the quantified scores to drive scheduling decisions, considering the current system load and other inference jobs on different models and input frames. SDRM3 has tunable parameters that provide fast adaptivity to dynamic workload changes based on a gradient descent-like online optimization, which typically converges within five steps for new workloads. In addition, we also propose a method to exploit model level dynamicity based on Supernet for exploiting the trade-off between the scheduling effectiveness and model performance (e.g., accuracy), which dynamically selects a proper sub-network in a Supernet based on the system loads. In our evaluation on five realistic RTMM workload scenarios, SDRM3 reduces the overall UXCost, which is a energy-delay-product (EDP)-equivalent metric for real-time applications defined in the paper, by 37.7% and 53.2% on geometric mean (up to 97.6% and 97.1%) compared to state-of-the-art baselines, which shows the efficacy of our scheduling methodology.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search2019
- SkipNet: Learning Dynamic Routing in Convolutional Networks2018
- BlockDrop: Dynamic Inference Paths in Residual Networks2018
- Temporal Convolutional Networks for Action Segmentation and Detection2017
- Overload management in real-time control applications using (m, k)-firm guaranteeIEEE Transactions on Parallel and Distributed Systems, 1999
- Combined task and message scheduling in distributed real-time systemsIEEE Transactions on Parallel and Distributed Systems, 1999
- Allocation and scheduling of precedence-related periodic tasksIEEE Transactions on Parallel and Distributed Systems, 1995
- New strategies for assigning real-time tasks to multiprocessor systemsIEEE Transactions on Computers, 1995
- The deferrable server algorithm for enhanced aperiodic responsiveness in hard real-time environmentsIEEE Transactions on Computers, 1995
- Scheduling processes with release times, deadlines, precedence and exclusion relationsIEEE Transactions on Software Engineering, 1990