SDTP: Semantic-Aware Decoupled Transformer Pyramid for Dense Image Prediction
- 24 March 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (9) , 6160-6173
Abstract
Although transformer has achieved great progress on computer vision tasks, the scale variation in dense image prediction is still the key challenge. Few effective multi-scale techniques are applied in transformer and there are two main limitations in the current methods. On the one hand, self-attention module in vanilla transformer fails to sufficiently exploit the diversity of semantic information because of its rigid mechanism. On the other hand, it is difficult to build attention and interaction among different levels due to the heavy computational burden. To alleviate this problem, we first revisit multi-scale problem in dense prediction, verifying the significance of diverse semantic representation and multi-scale interaction, and exploring the adaptation of transformer to pyramidal structure. Inspired by these findings, we propose a novel Semantic-aware Decoupled Transformer Pyramid (SDTP) for dense image prediction, consisting of Intra-level Semantic Promotion (ISP), Cross-level Decoupled Interaction (CDI) and Attention Refinement Function (ARF). ISP explores the semantic diversity in different receptive space through more flexible self-attention strategy. CDI builds the global attention and interaction among different levels in decoupled space which also solves the problem of heavy computation. Besides, ARF is further added to refine the attention in transformer. Experimental results demonstrate the validity and generality of the proposed method, which outperforms the state-of-the-art by a significant margin in dense image prediction tasks. Furthermore, the proposed components are all plug-and-play, which can be embedded in other methods.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2020AAA0106800)
- Natural Science Foundation of China (61902401, 62192785, 61972071, U1936204, 62122086, 62036011, 62192782, 61721004)
- Beijing Natural Science Foundation (M22005)
- Chinese Academy of Sciences (CAS) Key Research Program of Frontier Sciences (QYZDJ-SSW-JSC040)
- China Postdoctoral Science Foundation (2021M693402)
- Youth Innovation Promotion Association, CAS
This publication has 50 references indexed in Scilit:
- Deep Watershed Transform for Instance Segmentation2017
- Feature Pyramid Networks for Object Detection2017
- Fully Convolutional Instance-Aware Semantic Segmentation2017
- Scene Parsing through ADE20K Dataset2017
- InstanceCut: From Edges to Instances with MultiCut2017
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Instance-Aware Semantic Segmentation via Multi-task Network Cascades2016
- You Only Look Once: Unified, Real-Time Object Detection2016
- Fully convolutional networks for semantic segmentation2015
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998