Efficient Multitask Dense Predictor via Binarization
Preprint
Abstract
Multi-task learning for dense prediction has emerged as a pivotal area in computer vision, enabling simultaneous processing of diverse yet interrelated pixel-wise prediction tasks. However, the substantial computational demands of state-of-the-art (SoTA) models often limit their widespread deployment. This paper addresses this challenge by introducing network binarization to compress resource-intensive multi-task dense predictors. Specifically, our goal is to significantly accelerate multi-task dense prediction models via Binary Neural Networks (BNNs) while maintaining and even improving model performance at the same time. To reach this goal, we propose a Binary Multi-task Dense Predictor, Bi-MTDP, and several variants of Bi-MTDP, in which a multi-task dense predictor is constructed via specified binarized modules. Our systematical analysis of this predictor reveals that performance drop from binarization is primarily caused by severe information degradation. To address this issue, we introduce a deep information bottleneck layer that enforces representations for downstream tasks satisfying Gaussian distribution in forward propagation. Moreover, we introduce a knowledge distillation mechanism to correct the direction of information flow in backward propagation. Intriguingly, one variant of Bi-MTDP outperforms full-precision (FP) multi-task dense prediction SoTAs, ARTC (CNN-based) and InvPT (ViT-Based). This result indicates that Bi-MTDP is not merely a naive trade-off between performance and efficiency, but is rather a benefit of the redundant information flow thanks to the multi-task architecture. Code is available at https://github.com/42Shawn/BiMTDP.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Inverted Pyramid Multi-task Transformer for Dense Scene Understanding2022
- ProxyBNN: Learning Binarized Neural Networks via Proxy Matrices2020
- ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions2020
- MTI-Net: Multi-scale Task Interaction Networks for Multi-task Learning2020
- Bi-Real Net: Binarizing Deep Network Towards Real-Network PerformanceInternational Journal of Computer Vision, 2019
- Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics2018
- PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing2018
- Fully-Adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification2017
- UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory2017
- Learning Rich Features from RGB-D Images for Object Detection and Segmentation2014