CNNPC: End-Edge-Cloud Collaborative CNN Inference With Joint Model Partition and Compression
- 26 May 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (12) , 4039-4056
Abstract
Edge Intelligence (EI) aims at addressing concerns like response latency risen by the conflict between predominating Cloud-based deployments of computationally intensive AI applications and the expensive uploading of explosive end data. Convolutional Neural Networks (CNNs) leading the latest flourish of AI inevitably suffer from the aforementioned conflict. There emerge increasing EI-driven attempts on fast CNN inference with high accuracy in the End-Edge-Cloud (EEC) collaborative computing paradigm, where, however, neither model compression approaches for on-device inference nor collaborative inference methods across devices can effectively achieve the trade-off between latency and accuracy of End-to-End (E2E) inference. In this article, we present CNNPC that jointly partitions and compresses CNNs for fast inference with high accuracy in collaborative EEC systems. We implemented CNNPC (source code available at https://github.com/IoTDATALab/CNNPC ) and evaluated its performance within extensive real-world EEC scenarios. Experimental results demonstrate that, compared with state-of-the-art single-end and collaborative approaches, without obvious accuracy loss, collaborative inference based on CNNPC is up to $1.6\times$ and $5.6\times$ faster, and requires as low as $4.30\%$ and $6.48\%$ communications, respectively. Besides, when determines the optimal strategy, CNNPC requires as low as $0.1\%$ actual compression operations that the traversal method (the only viable method providing the theoretically optimal strategy) requires.
Keywords
Funding Information
- National Key Research and Development Program of China (2020YFA0713900)
- National Natural Science Foundation of China (61772410, 61802298, 62172329, U1811461, U21A6005, 11690011)
- China Postdoctoral Science Foundation (2020T130513, 2019M663726)
This publication has 41 references indexed in Scilit:
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Taking the Human Out of the Loop: A Review of Bayesian OptimizationProceedings of the IEEE, 2015
- DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving2015
- Going deeper with convolutions2015
- Image Super-Resolution Using Deep Convolutional NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Deep learningNature, 2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Scaling Distributed Machine Learning with the Parameter Server2014
- Towards wearable cognitive assistance2014
- Deep Hierarchies in the Primate Visual Cortex: What Can We Learn for Computer Vision?IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012