Stream-ViT: Learning Streamlined Convolutions in Vision Transformer
- 27 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 3755-3765
Abstract
Recently Vision Transformer (ViT) and Convolution Neural Network (CNN) start to emerge as a hybrid deep architecture with better model capacity, generalization, and latency trade-off. Most of these hybrid architectures often directly stack self-attention module with static convolution or fuse their outputs through two pathways within each block. Instead, we present a new Transformer architecture (namely Stream-ViT) to novelly integrate ViT with streamlined convolutions, i.e., a series of high-to-low resolution convolutions. The kernels of each convolution are dynamically learnt on a basis of current input features plus pre-learnt kernels throughout the whole network. The new architecture incorporates a critical pathway to streamline kernel generation that triggers the interactions between dynamically learnt convolutions across different layers. Moreover, the introduction of a layer-wise streamlined convolution is functionally equivalent to a squeezed version of multi-branch convolution structure, thereby improving the capacity of self-attention module with enlarged cardinality in a cost-efficient manner. We validate the superiority of Stream-ViT over multiple vision tasks, and its performances surpass state-of-the-art ViT and CNN backbones with comparable FLOPs.Keywords
Funding Information
- Beijing Municipal Science and Technology (Z241100001324002)
- Beijing Nova Program (20240484681)
This publication has 60 references indexed in Scilit:
- ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and BeyondInternational Journal of Computer Vision, 2023
- DeiT III: Revenge of the ViT2022
- Dynamic Temporal Filtering in Video Models2022
- Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning2022
- PVT v2: Improved baselines with pyramid vision transformerComputational Visual Media, 2022
- WeightNet: Revisiting the Design Space of Weight Networks2020
- End-to-End Object Detection with Transformers2020
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- Unified Perceptual Parsing for Scene Understanding2018
- Microsoft COCO: Common Objects in Context2014