Precision-aware Latency and Energy Balancing on Multi-Accelerator Platforms for DNN Inference
Authors:
- 7 August 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Abstract
The need to execute Deep Neural Networks (DNNs) at low latency and low power at the edge has spurred the development of new heterogeneous Systems-on-Chips (SoCs) encapsulating a diverse set of hardware accelerators. How to optimally map a DNN onto such multi-accelerator systems is an open problem. We propose ODiMO, a hardware-aware tool that performs a fine-grain mapping across different accelerators on-chip, splitting individual layers and executing them in parallel, to reduce inference energy consumption or latency, while taking into account each accelerator's quantization precision to maintain accuracy. Pareto-optimal networks in the accuracy vs. energy or latency space are pursued for three popular dataset/DNN pairs, and deployed on the DIANA heterogeneous ultra-low power edge AI SoC. We show that ODiMO reduces energy/latency by up to 33%/31% with limited accuracy drop (−0.53%/-0.32%) compared to manual heuristic mappings.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes2022
- The Best of Many Worlds: Scheduling Machine Learning Inference on CPU-GPU Integrated Architectures2022
- AI Accelerator Survey and Trends2021
- Rethinking Differentiable Search for Mixed-Precision Neural Networks2020
- AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators2020
- A Power Efficient Neural Network Implementation on Heterogeneous FPGA and GPU Devices2019
- HAQ: Hardware-Aware Automated Quantization With Mixed Precision2019
- Automatic Mapping for OpenCL-Programs on CPU/GPU Heterogeneous Platforms2018
- Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference2018
- Deep Residual Learning for Image Recognition2016