Model Parallelism Optimization for Distributed Inference via Decoupled CNN Structure
- 1 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. PP (10459219) , 1
Abstract
In this paper, we propose DeCNN, a more effective inference approach that uses decoupled CNN structure to optimize model parallelism for distributed inference on end-user devices. DeCNN is novel consisting of three schemes. Scheme-1 is structure-level optimization. It exploits group convolution and channel shuffle to decouple the original CNN structure for model parallelism. Scheme-2 is partition-level optimization. It is based on channel group to partition the convolutional layers, and then leverages input-based method to partition the fully connected layers, further exposing high degree of parallelism. Scheme-3 is communication-level optimization. It uses inter-sample parallelism to hide communications for better performance and robustness, especially in the weak network connections. We use ImageNet to evaluate the effectiveness of DeCNN for image classification on a distributed multi-ARM platform. Notably, when using the number of devices from 1 to 4, DeCNN can accelerate the inference of large-scale ResNet50 by 3.21x, and reduce 65.3% memory footprint, with 1.29% accuracy improvement.Keywords
Funding Information
- National Natural Science Foundation of China (61802446)
This publication has 21 references indexed in Scilit:
- Convolutional Neural Network-Based Embarrassing Situation Detection under Camera for Social Robot in Smart HomesSensors, 2018
- In-Situ AI: Towards Autonomous and Incremental Deep Learning for IoT Systems2018
- Towards Reasoning VehiclesACM Computing Surveys, 2017
- Scaling deep learning on GPU and knights landing clusters2017
- Channel Pruning for Accelerating Very Deep Neural Networks2017
- MoDNN: Local distributed mobile computing system for Deep Neural Network2017
- Characterizing and Improving WiFi Latency in Large-Scale Operational Networks2016
- Deep Residual Learning for Image Recognition2016
- An Early Resource Characterization of Deep Learning on Wearables, Smartphones and Internet-of-Things Devices2015
- ImageNet: A large-scale hierarchical image database2009