OmniBoost: Boosting Throughput of Heterogeneous Embedded Devices under Multi-DNN Workload
Open Access
- 9 July 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
Modern Deep Neural Networks (DNNs) exhibit profound efficiency and accuracy properties. This has introduced application workloads that comprise of multiple DNN applications, raising new challenges regarding workload distribution. Equipped with a diverse set of accelerators, newer embedded system present architectural heterogeneity, which current run-time controllers are unable to fully utilize. To enable high throughput in multi-DNN workloads, such a controller is ought to explore hundreds of thousands of possible solutions to exploit the underlying heterogeneity. In this paper, we propose OmniBoost, a lightweight and extensible multi-DNN manager for heterogeneous embedded devices. We leverage stochastic space exploration and we combine it with a highly accurate performance estimator to observe a ×4.6 average throughput boost compared to other state-of-the-art methods. The evaluation was performed on the HiKey970 development board. Our code is publicly available at https://github.com/AndreasKaratzas/omniboost-v1.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- MOSAIC: Heterogeneity-, Communication-, and Constraint-Aware Model Slicing and Execution for Accurate and Efficient Inference2019
- The Case for Exploiting Underutilized Resources in Heterogeneous Mobile Architectures2019
- Machine Learning at Facebook: Understanding Inference at the Edge2019
- RSTensorFlow2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- CNNdroid2016
- Deep Residual Learning for Image Recognition2016
- Rethinking the Inception Architecture for Computer Vision2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Cognitron: A self-organizing multilayered neural networkBiological Cybernetics, 1975