SDPT: Semantic-Aware Dimension-Pooling Transformer for Image Segmentation
Open Access
- 3 July 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 25 (11) , 15934-15946
Abstract
Image segmentation plays a critical role in autonomous driving by providing vehicles with a detailed and accurate understanding of their surroundings. Transformers have recently shown encouraging results in image segmentation. However, transformer-based models are challenging to strike a better balance between performance and efficiency. The computational complexity of the transformer-based models is quadratic with the number of inputs, which severely hinders their application in dense prediction tasks. In this paper, we present the semantic-aware dimension-pooling transformer (SDPT) to mitigate the conflict between accuracy and efficiency. The proposed model comprises an efficient transformer encoder for generating hierarchical features and a semantic-balanced decoder for predicting semantic masks. In the encoder, a dimension-pooling mechanism is used in the multi-head self-attention (MHSA) to reduce the computational cost, and a parallel depth-wise convolution is used to capture local semantics. Simultaneously, we further apply this dimension-pooling attention (DPA) to the decoder as a refinement module to integrate multi-level features. With such a simple yet powerful encoder-decoder framework, we empirically demonstrate that the proposed SDPT achieves excellent performance and efficiency on various popular benchmarks, including ADE20K, Cityscapes, and COCO-Stuff. For example, our SDPT achieves 48.6 $\%$ mIOU on the ADE20K dataset, which outperforms the current methods with fewer computational costs. The codes can be found at https://github.com/HuCaoFighting/SDPT.Keywords
Funding Information
- National Natural Science Foundation of China (62372329)
- Shanghai Scientific Innovation Foundation (23DZ1203400)
- Shanghai Rising Star Program (21QC1400900)
- Tongji–Qomolo Autonomous Driving Commercial Vehicle Joint Laboratory Project
- Xiaomi Young Talents Program
This publication has 70 references indexed in Scilit:
- A Transformer-Based Decoder for Semantic Segmentation with Multi-level Context Mining2022
- PVT v2: Improved baselines with pyramid vision transformerComputational Visual Media, 2022
- BiSeNet V2: Bilateral Network with Guided Aggregation for Real-Time Semantic SegmentationInternational Journal of Computer Vision, 2021
- Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistanceInternational Journal of Computer Assisted Radiology and Surgery, 2021
- A review of deep learning methods for semantic segmentation of remote sensing imageryExpert Systems with Applications, 2021
- Human segmentation in surveillance video with deep learningMultimedia Tools and Applications, 2020
- ICNet for Real-Time Semantic Segmentation on High-Resolution Images2018
- BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation2018
- A Survey: Image Segmentation Techniques2018
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015