Collaborative Inference Acceleration with Non-Penetrative Tensor Partitioning
Authors:
Preprint
Abstract
The inference of large-sized images on Internet of Things (IoT) devices is commonly hindered by limited resources, while there are often stringent latency requirements for Deep Neural Network (DNN) inference. Currently, this problem is generally addressed by collaborative inference, where the large-sized image is partitioned into multiple tiles, and each tile is assigned to an IoT device for processing. However, since significant latency will be incurred due to the communication overhead caused by tile sharing, the existing collaborative inference strategy is inefficient for convolutional computation, which is indispensable for any DNN. To reduce it, we propose Non-Penetrative Tensor Partitioning (NPTP), a fine-grained tensor partitioning method that reduces the communication latency by minimizing the communication load of tiles shared, thereby reducing inference latency. We evaluate NPTP with four widely-adopted DNN models. Experimental results demonstrate that NPTP achieves a 1.44-1.68x inference speedup relative to CoEdge, a state-of-the-art (SOTA) collaborative inference algorithm.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Enabling ImageNet-Scale Deep Learning on MCUs for Accurate and Efficient InferenceIEEE Internet of Things Journal, 2023
- Stochastic Cumulative DNN Inference With RL-Aided Adaptive IoT Device-Edge CollaborationIEEE Internet of Things Journal, 2023
- A Survey on Collaborative DNN Inference for Edge IntelligenceMachine Intelligence Research, 2023
- AI for UAV-Assisted IoT Applications: A Comprehensive ReviewIEEE Internet of Things Journal, 2023
- Internet of things for smart factories in industry 4.0, a reviewInternet of Things and Cyber-Physical Systems, 2023
- Collaborative Inference for AI-Empowered IoT DevicesIEEE Internet of Things Magazine, 2022
- Toward Decentralized and Collaborative Deep Learning Inference for Intelligent IoT DevicesIEEE Network, 2022
- Arrhythmia diagnosis of young martial arts athletes based on deep learning for smart medical careNeural Computing and Applications, 2021
- Toward Collaborative Inferencing of Deep Neural Networks on Internet-of-Things DevicesIEEE Internet of Things Journal, 2020
- IoT-Enabled Real-Time Production Performance Analysis and Exception Diagnosis ModelIEEE Transactions on Automation Science and Engineering, 2015