Fuse & Calibrate: A bi-directional Vision-Language Guided Framework for Referring Image Segmentation
Authors:
- Yichen YanYYYichen Yan
- Xingjian HeXHXingjian He
- Shichen LuSLShichen Lu
Preprint
Abstract
Referring Image Segmentation (RIS) aims to segment an object described in natural language from an image, with the main challenge being a text-to-pixel correlation. Previous methods typically rely on single-modality features, such as vision or language features, to guide the multi-modal fusion process. However, this approach limits the interaction between vision and language, leading to a lack of fine-grained correlation between the language description and pixel-level details during the decoding process. In this paper, we introduce FCNet, a framework that employs a bi-directional guided fusion approach where both vision and language play guiding roles. Specifically, we use a vision-guided approach to conduct initial multi-modal fusion, obtaining multi-modal features that focus on key vision information. We then propose a language-guided calibration module to further calibrate these multi-modal features, ensuring they understand the context of the input sentence. This bi-directional vision-language guided approach produces higher-quality multi-modal features sent to the decoder, facilitating adaptive propagation of fine-grained semantic information from textual features to visual features. Experiments on RefCOCO, RefCOCO+, and G-Ref datasets with various backbones consistently show our approach outperforming state-of-the-art methods.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- Parallel Vertex Diffusion for Unified Visual GroundingProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Toward Robust Referring Image SegmentationIEEE Transactions on Image Processing, 2024
- Cross-Aware Early Fusion With Stage-Divided Vision and Language Transformer Encoders for Referring Image SegmentationIEEE Transactions on Multimedia, 2023
- VLT: Vision-Language Transformer and Query Generation for Referring SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019