Language-Aware Spatial-Temporal Collaboration for Referring Video Segmentation
- 10 January 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-14
Abstract
Given a natural language referring expression, the goal of referring video segmentation task is to predict the segmentation mask of the referred object in the video. Previous methods only adopt 3D CNNs upon the video clip as a single encoder to extract a mixed spatio-temporal feature for the target frame. Though 3D convolutions are able to recognize which object is performing the described actions, they still introduce misaligned spatial information from adjacent frames, which inevitably confuses features of the target frame and leads to inaccurate segmentation. To tackle this issue, we propose a language-aware spatial-temporal collaboration framework that contains a 3D temporal encoder upon the video clip to recognize the described actions, and a 2D spatial encoder upon the target frame to provide undisturbed spatial features of the referred object. For multimodal features extraction, we propose a Cross-Modal Adaptive Modulation (CMAM) module and its improved version CMAM+ to conduct adaptive cross-modal interaction in the encoders with spatial- or temporal-relevant language features which are also updated progressively to enrich linguistic global context. In addition, we also propose a Language-Aware Semantic Propagation (LASP) module in the decoder to propagate semantic information from deep stages to the shallow stages with language-aware sampling and assignment, which is able to highlight language-compatible foreground visual features and suppress language-incompatible background visual features for better facilitating the spatial-temporal collaboration. Extensive experiments on four popular referring video segmentation benchmarks demonstrate the superiority of our method over the previous state-of-the-art methods. Code is available at https://github.com/RyanHTR/CSTMKeywords
All Related Versions
This publication has 60 references indexed in Scilit:
- Segmentation from Natural Language Expressions2016
- Modeling Context Between Objects for Referring Expression Understanding2016
- Deep Residual Learning for Image Recognition2016
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Can humans fly? Action understanding with multiple classes of actors2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Glove: Global Vectors for Word Representation2014
- Towards Understanding Action Recognition2013