DSCAFormer: Lightweight Vision Transformer With Dual-Branch Spatial Channel Aggregation
Authors:
Open Access
- 28 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 75272-75288
Abstract
Vision Transformer (ViT) models with strong performance are widely popular in the vision field. However, they are often limited by computation and model size, making the study of lightweight models critical. Existing lightweight ViT models often exhibit a weak dependence on high-frequency information and lack an effective multi-frequency fusion mechanism. This limitation typically necessitates the use of complex computational resources and the expansion of model scale to compensate for these shortcomings. In response to this situation, this paper proposes the DSCAFormer model, which develops a novel architecture that includes a local convolutional block with attention-like algorithms and a global spatial transformer block. These components use a channel splitting strategy with a gating mechanism to merge local and global information, forming a multi-frequency spatial feature extractor capable of effectively integrating both local and global information from images.Furthermore, a channel aggregation method is introduced to enhance the extraction of spatial information within the channel space, thereby enabling spatial feature perception within the context and the allocation of multi-feature computation. A series of comprehensive experiments were conducted in the domains of classification, detection, and instance segmentation, which demonstrated the DSCAFormer model’s remarkable scalability and competitiveness. For instance, on the ImageNet-1K dataset, the DSCAFormer model, which utilized 2.5M, 4.3M, and 7.4M parameters, achieved top-1 accuracies of 72.5%, 76.7%, and 79.5%, respectively, with 0.4 G, 0.7G, and 1.2G FLOPs.These results outperform the MobileViTv2-0.5/0.75/1.0 models, with respective accuracy improvements of 2.3%, 1.1%, and 1.4%, and reductions in FLOPs of 20%, 30%, and 33%. In addition, DSCAFormer has also shown competitive performance in downstream tasks.Keywords
Funding Information
- Natural Science Foundation of Zhejiang Province (LY21F010016)
- Natural Science Foundation of Zhejiang Sci-Tech University (19022397-Y)
This publication has 46 references indexed in Scilit:
- Scene Text Segmentation via Multi-Task Cascade Transformer With Paired Data SynthesisIEEE Access, 2023
- EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications2023
- Exploring Plain Vision Transformer Backbones for Object Detection2022
- EdgeViTs: Competing Light-Weight CNNs on Mobile Devices with Vision Transformers2022
- ParC-Net: Position Aware Circular Convolution with Merits from ConvNets and Transformer2022
- PVT v2: Improved baselines with pyramid vision transformerComputational Visual Media, 2022
- End-to-End Object Detection with Transformers2020
- Sigmoid-weighted linear units for neural network function approximation in reinforcement learningNeural Networks, 2018
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018
- Microsoft COCO: Common Objects in Context2014