Latency-Aware Unified Dynamic Networks for Efficient Image Recognition
- 25 April 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (12) , 7760-7774
Abstract
Dynamic computation has emerged as a promising strategy to improve the inference efficiency of deep networks. It allows selective activation of various computing units, such as layers or convolution channels, or adaptive allocation of computation to highly informative spatial regions in image features, thus significantly reducing unnecessary computations conditioned on each input sample. However, the practical efficiency of dynamic models does not always correspond to theoretical outcomes. This discrepancy stems from three key challenges: 1) The absence of a unified formulation for various dynamic inference paradigms, owing to the fragmented research landscape; 2) The undue emphasis on algorithm design while neglecting scheduling strategies , which are critical for optimizing computational performance and resource utilization in CUDA-enabled GPU settings; and 3) The cumbersome process of evaluating practical latency, as most existing libraries are tailored for static operators. To address these issues, we introduce Latency-Aware Unified Dynamic Networks (LAUDNet) , a comprehensive framework that amalgamates three cornerstone dynamic paradigms-spatially-adaptive computation, dynamic layer skipping, and dynamic channel skipping-under a unified formulation. To reconcile theoretical and practical efficiency, LAUDNet integrates algorithmic design with scheduling optimization, assisted by a latency predictor that accurately and efficiently gauges the inference latency of dynamic operators. This latency predictor harmonizes considerations of algorithms, scheduling strategies, and hardware attributes. We empirically validate various dynamic paradigms within the LAUDNet framework across a range of vision tasks, including image classification, object detection, and instance segmentation. Our experiments confirm that LAUDNet effectively narrows the gap between theoretical and real-world efficiency. For example, LAUDNet can reduce the practical latency of its static counterpart, ResNet-101, by over 50% on hardware platforms such as V100, RTX3090, and TX2 GPUs. Furthermore, LAUDNet surpasses competing methods in the trade-off between accuracy and efficiency. Code is available at: https://www.github.com/LeapLabTHU/LAUDNet .Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2021ZD0140407)
- National Natural Science Foundation of China (42327901, 62321005)
- Guoqiang Institute, Tsinghua University
- High-Flyer AI
This publication has 42 references indexed in Scilit:
- Learning to Weight Samples for Dynamic Early-Exiting Networks2022
- PTQ4ViT: Post-training Quantization for Vision Transformers with Twin Uniform Quantization2022
- Channel Selection Using Gumbel Softmax2020
- Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation2020
- End-to-End Object Detection with Transformers2020
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018
- Convolutional Networks with Adaptive Inference Graphs2018
- SkipNet: Learning Dynamic Routing in Convolutional Networks2018
- Microsoft COCO: Common Objects in Context2014
- ImageNet: A large-scale hierarchical image database2009