Latency-aware Unified Dynamic Networks for Efficient Image Recognition
Authors:
- Yizeng HanYHYizeng Han
- Zeyu LiuZLZeyu Liu
- Zhihang YuanZYZhihang Yuan
- Yifan PuYPYifan Pu
- Chaofei WangCWChaofei Wang
- Shiji SongSSShiji Song
- Gao HuangGHGao Huang
Preprint
Abstract
Dynamic computation has emerged as a promising avenue to enhance the inference efficiency of deep networks. It allows selective activation of computational units, leading to a reduction in unnecessary computations for each input sample. However, the actual efficiency of these dynamic models can deviate from theoretical predictions. This mismatch arises from: 1) the lack of a unified approach due to fragmented research; 2) the focus on algorithm design over critical scheduling strategies, especially in CUDA-enabled GPU contexts; and 3) challenges in measuring practical latency, given that most libraries cater to static operations. Addressing these issues, we unveil the Latency-Aware Unified Dynamic Networks (LAUDNet), a framework that integrates three primary dynamic paradigms-spatially adaptive computation, dynamic layer skipping, and dynamic channel skipping. To bridge the theoretical and practical efficiency gap, LAUDNet merges algorithmic design with scheduling optimization, guided by a latency predictor that accurately gauges dynamic operator latency. We've tested LAUDNet across multiple vision tasks, demonstrating its capacity to notably reduce the latency of models like ResNet-101 by over 50% on platforms such as V100, RTX3090, and TX2 GPUs. Notably, LAUDNet stands out in balancing accuracy and efficiency. Code is available at: https://www.github.com/LeapLabTHU/LAUDNet.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Learning to Weight Samples for Dynamic Early-Exiting Networks2022
- PTQ4ViT: Post-training Quantization for Vision Transformers with Twin Uniform Quantization2022
- Channel Selection Using Gumbel Softmax2020
- Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation2020
- Convolutional Networks with Adaptive Inference GraphsInternational Journal of Computer Vision, 2019
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018
- SkipNet: Learning Dynamic Routing in Convolutional Networks2018
- More is Less: A More Complicated Network with Less Inference Complexity2017
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014