Hierarchical Co-Attention Propagation Network for Zero-Shot Video Object Segmentation
- 19 April 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 2348-2359
Abstract
Zero-shot video object segmentation (ZS-VOS) aims to segment foreground objects in a video sequence without prior knowledge of these objects. However, existing ZS-VOS methods often struggle to distinguish between foreground and background or to keep track of the foreground in complex scenarios. The common practice of introducing motion information, such as optical flow, can lead to overreliance on optical flow estimation. To address these challenges, we propose an encoder-decoder-based hierarchical co-attention propagation network (HCPN) capable of tracking and segmenting objects. Specifically, our model is built upon multiple collaborative evolutions of the parallel co-attention module (PCM) and the cross co-attention module (CCM). PCM captures common foreground regions among adjacent appearance and motion features, while CCM further exploits and fuses cross-modal motion features returned by PCM. Our method is progressively trained to achieve hierarchical spatio-temporal feature propagation across the entire video. Experimental results demonstrate that our HCPN outperforms all previous methods on public benchmarks, showcasing its effectiveness for ZS-VOS. Code and pre-trained model can be found at https://github.com/NUST-Machine-Intelligence-Laboratory/HCPN.Keywords
All Related Versions
Funding Information
- Natural Science Foundation of Jiangsu Province (BK20210327, BK20220934, BK20220938, BK20220936)
- Jiangsu Funding Program for Excellent Postdoctoral Talent (2022ZB267, 2022ZB271)
- National Natural Science Foundation of China (62102182, 62202227)
- China Postdoctoral Science Foundation (2022M721626, 2022M721628, 2022M711635)
This publication has 79 references indexed in Scilit:
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Deep Residual Learning for Image Recognition2016
- A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation2016
- Attention to Scale: Scale-Aware Semantic Image Segmentation2016
- Learning to segment moving objects in videos2015
- Global Contrast Based Salient Region DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- Microsoft COCO: Common Objects in Context2014
- Segmentation of Moving Objects by Long Term Video AnalysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Fast Object Segmentation in Unconstrained Video2013
- Learning object class detectors from weakly annotated video2012