Memory- and Communication-Aware Model Compression for Distributed Deep Learning Inference on IoT
Authors:
Open Access
- 8 October 2019
- journal article
- conference paper
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Embedded Computing Systems
- Vol. 18 (5s) , 1-22
Abstract
Model compression has emerged as an important area of research for deploying deep learning models on Internet-of-Things (IoT). However, for extremely memory-constrained scenarios, even the compressed models cannot fit within the memory of a single device and, as a result, must be distributed across multiple devices. This leads to a distributed inference paradigm in which memory and communication costs represent a major bottleneck. Yet, existing model compression techniques are not communication-aware. Therefore, we propose Network of Neural Networks (NoNN), a new distributed IoT learning paradigm that compresses a large pretrained ‘teacher’ deep network into several disjoint and highly-compressed ‘student’ modules, without loss of accuracy. Moreover, we propose a network science-based knowledge partitioning algorithm for the teacher model, and then train individual students on the resulting disjoint partitions. Extensive experimentation on five image classification datasets, for user-defined memory/performance budgets, show that NoNN achieves higher accuracy than several baselines and similar accuracy as the teacher model, while using minimal communication among students. Finally, as a case study, we deploy the proposed model for CIFAR-10 dataset on edge devices and demonstrate significant improvements in memory footprint (up to 24×), performance (up to 12×), and energy per node (up to 14×) compared to the large teacher model. We further show that for distributed inference on multiple edge devices, our proposed NoNN model results in up to 33× reduction in total latency w.r.t. a state-of-the-art model compression baseline.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge ClustersIEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2018
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- MeDNN: A distributed mobile system with enhanced partition and deployment for large-scale DNNs2017
- Recurrent neural network training with dark knowledge transfer2016
- Recognizing indoor scenes2009
- Modularity and community structure in networksProceedings of the National Academy of Sciences, 2006