HiDP: Hierarchical DNN Partitioning for Distributed Inference on Heterogeneous Edge Platforms
Preprint
Abstract
Edge inference techniques partition and distribute Deep Neural Network (DNN) inference tasks among multiple edge nodes for low latency inference, without considering the core-level heterogeneity of edge nodes. Further, default DNN inference frameworks also do not fully utilize the resources of heterogeneous edge nodes, resulting in higher inference latency. In this work, we propose a hierarchical DNN partitioning strategy (HiDP) for distributed inference on heterogeneous edge nodes. Our strategy hierarchically partitions DNN workloads at both global and local levels by considering the core-level heterogeneity of edge nodes. We evaluated our proposed HiDP strategy against relevant distributed inference techniques over widely used DNN models on commercial edge devices. On average our strategy achieved 38% lower latency, 46% lower energy, and 56% higher throughput in comparison with other relevant approaches.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- PArtNNer: Platform-Agnostic Adaptive Edge-Cloud DNN Partitioning for Minimizing End-to-End LatencyACM Transactions on Embedded Computing Systems, 2024
- HASP: Hierarchical Asynchronous Parallelism for Multi-NN TasksIEEE Transactions on Computers, 2023
- DISNET: Distributed Micro-Split Deep Learning in Heterogeneous Dynamic IoTIEEE Internet of Things Journal, 2023
- Automated Exploration and Implementation of Distributed CNN Inference at the EdgeIEEE Internet of Things Journal, 2023
- Energy-Aware Scenario-Based Mapping of Deep Learning Applications Onto Heterogeneous Processors Under Real-Time ConstraintsIEEE Transactions on Computers, 2022
- Enabling DNN Acceleration With Data and Model Parallelization Over Ubiquitous End DevicesIEEE Internet of Things Journal, 2021
- CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge DevicesIEEE/ACM Transactions on Networking, 2020
- High-Throughput CNN Inference on Embedded ARM Big.LITTLE Multicore ProcessorsIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2019