Online Resource Provisioning and Batch Scheduling for AIoT Inference Serving in an XPU Edge Cloud
- 30 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computing
- Vol. 13 (1) , 234-249
Abstract
Driven by the accelerated convergence of artificial intelligence (AI) and the Internet of Things (IoT), the recent years have witnessed the booming of Artificial Intelligence of Things (AIoT). Edge clouds place computing and service capabilities at the network edges to reduce network transmission overhead, which has been widely recognized as the critical infrastructure for AIoT applications. Meanwhile, to accelerate computationintensive edge cloud AI operations, specialized AI accelerators such as GPU, NPU, and TPU have been increasingly integrated into edge clouds. For such emerging XPU edge clouds, utilizing costly XPUs more efficiently has become a significant challenge. In this paper, we present an online optimization framework for joint resource provisioning and batch scheduling for more costefficient AIoT inference serving in an XPU edge cloud. The essential optimization process for the online framework is to first adaptively batch inference tasks to increase the system throughput without compromising the service level agreement (SLA). Next, heterogeneous XPU resources are provisioned for the batches. Finally, the resource instance is consolidated to a minimum of physical servers. Via extensive trace-driven simulations, we verify the performance of the presented online optimization framework.Keywords
This publication has 35 references indexed in Scilit:
- Quasi-regulation and mismatch mitigation technique for switched tank converters2023
- Queueing analysis of GPU-based inference servers with dynamic batching: A closed-form characterizationPerformance Evaluation, 2021
- Evaluating and analyzing the energy efficiency of CNN inference on high‐performance GPUConcurrency and Computation: Practice and Experience, 2020
- A Survey of Accelerator Architectures for Deep Neural NetworksEngineering, 2020
- The Architectural Implications of Facebook's DNN-Based Personalized Recommendation2020
- A survey of deep learning techniques for autonomous drivingJournal of Field Robotics, 2019
- Batch Size Influence on Performance of Graphic and Tensor Processing Units During Training and Inference Phases2019
- Efficient Resource Management for Virtual Machine Allocation in Cloud Data Centers2018
- Vector bin packing with heterogeneous bins: application to the machine reassignment problemAnnals of Operations Research, 2015
- Enhanced First-Fit Decreasing Algorithm for Energy-Aware Job Scheduling in Cloud2014