UniTR: A Unified TRansformer-Based Framework for Co-Object and Multi-Modal Saliency Detection
- 26 February 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 7622-7635
Abstract
Recent years have witnessed a growing interest in co-object segmentation and multi-modal salient object detection. Many efforts are devoted to segmenting co-existed objects among a group of images or detecting salient objects from different modalities. Albeit the appreciable performance achieved on respective benchmarks, each of these methods is limited to a specific task and cannot be generalized to other tasks. In this paper, we develop a Uni fied TR ansformer-based framework, namely UniTR , aiming at tackling the above tasks individually with a unified architecture. Specifically, a transformer module (CoFormer) is introduced to learn the consistency of relevant objects or complementarity from different modalities. To generate high-quality segmentation maps, we adopt a dual-stream decoding paradigm that allows the extracted consistent or complementary information to better guide mask prediction. Moreover, a feature fusion module (ZoomFormer) is designed to enhance backbone features and capture multi-granularity and multi-semantic information. Extensive experiments show that our UniTR performs well on 17 benchmarks , and surpasses existing state-of-the-art approaches.Keywords
Funding Information
- National Natural Science Foundation of China (62371009, 61971008)
This publication has 123 references indexed in Scilit:
- Accurate RGB-D Salient Object Detection via Collaborative Learning2020
- Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection2020
- Learning Discriminative Feature with CRF for Unsupervised Video Object Segmentation2020
- Cross-Modal Weighting Network for RGB-D Salient Object Detection2020
- EfficientFCN: Holistically-Guided Decoding for Semantic Segmentation2020
- End-to-End Object Detection with Transformers2020
- Deep Object Co-segmentation2019
- Semantic Aware Attention Based Deep Object Co-segmentation2019
- Unsupervised Video Object Segmentation with Motion-Based Bilateral Networks2018
- Pyramid Dilated Deeper ConvLSTM for Video Salient Object Detection2018