TBDB: Token Bucket-Based Dynamic Batching for Resource Scheduling Supporting Neural Network Inference in Intelligent Consumer Electronics
- 5 December 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. 70 (1) , 1134-1144
Abstract
Consumer electronics such as mobile phones, wearable devices, and vehicle electronics use many intelligent applications such as voice commands, machine translation, and face recognition. These applications require large inference workloads to perform intelligent tasks, which are often completed using deep neural network (DNN) models. Traditional approaches rely on pure cloud computing, with consumer devices collecting data and cloud computing platforms completing inference tasks. In real life, the workloads of these applications are not fixed and are likely to exhibit fluctuations or unexpected surges, increasing the workload of cloud computing platforms. Simply increasing server resources often leads to resource waste. Therefore, a dynamic resource scheduling method is needed. In this paper, a token bucket-based dynamic batching (TBDB) algorithm that maintains throughput while reducing latency and increasing device utilization, especially for large volumes of requests, is proposed. Our work includes the following achievements: 1) We employ the token bucket algorithm to determine the workload, considering the concurrency and frequency of the data. We dynamically vary the maximum batch size (MBS) that will trigger the inference process for the next batch. 2) A low-coupling mode architecture that can be embedded into various consumer electronics in a plug-and-play manner is designed. 3) The performance of the electronic devices and the maximum latency are studied to provide guidance for setting hyperparameters. Finally, we evaluate the effectiveness of our method in three consumer electronic scenarios and present a theoretical analysis for setting hyperparameters in different scenarios.Keywords
Funding Information
- National Key Research and Development Program of China (2022YFF0902500)
This publication has 30 references indexed in Scilit:
- A distributed game theoretical approach for credibility-guaranteed multimedia data offloading in MECInformation Sciences, 2023
- Context-and category-aware double self-attention model for next POI recommendationApplied Intelligence, 2023
- A load-balancing problem for distributed bulk-service queues with size-dependent batch processing timesQueueing Systems, 2022
- Deep learning to mobile hypermedia and multimediaWireless Networks, 2022
- EAIS: Energy-aware adaptive scheduling for CNN inference on high-performance GPUsFuture Generation Computer Systems, 2022
- Gillis: Serving Large Neural Networks in Serverless Functions with Automatic Model Partitioning2021
- Lazy Batching: An SLA-aware Batching System for Cloud Machine Learning Inference2021
- Nanily: A QoS-Aware Scheduling for DNN Inference Workload in Clouds2019
- Trading Private Range Counting over Big IoT Data2019
- Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective2018