Real-time neural network inference on extremely weak devices
Open Access
- 14 October 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 200-213
Abstract
With the wide adoption of AI applications, there is a pressing need of enabling real-time neural network (NN) inference on small embedded devices, but deploying NNs and achieving high performance of NN inference on these small devices is challenging due to their extremely weak capabilities. Although NN partitioning and offloading can contribute to such deployment, they are incapable of minimizing the local costs at embedded devices. Instead, we suggest to address this challenge via agile NN offloading, which migrates the required computations in NN offloading from online inference to offline learning. In this paper, we present AgileNN, a new NN offloading technique that achieves real-time NN inference on weak embedded devices by leveraging eXplainable AI techniques, so as to explicitly enforce feature sparsity during the training phase and minimize the online computation and communication costs. Experiment results show that AgileNN's inference latency is >6X lower than the existing schemes, ensuring that sensory data on embedded devices can be timely consumed. It also reduces the local device's resource consumption by >8X, without impairing the inference accuracy.Keywords
All Related Versions
Funding Information
- National Science Foundation (CNS-1812407, CNS-2029520, IIS-1956002, IIS-2205360, CCF-2217003 and CCF-2215042)
This publication has 38 references indexed in Scilit:
- Neurosurgeon2017
- Inter-Technology Backscatter2016
- "Why Should I Trust You?"2016
- Deep Residual Learning for Image Recognition2016
- When Face Recognition Meets with Deep Learning: An Evaluation of Convolutional Neural Networks for Face Recognition2015
- Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification2015
- An improved deep learning architecture for person re-identification2015
- Optimal sampling frequency and bias error modeling for foot-mounted IMUs2013
- The JPEG still picture compression standardIEEE Transactions on Consumer Electronics, 1992
- MPEGCommunications of the ACM, 1991