Semi-Supervised Video Object Segmentation via Learning Object-Aware Global-Local Correspondence
- 19 July 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 32 (12) , 8153-8164
Abstract
In semi-supervised video object segmentation (VOS) task, temporal coherent object-level cues play a key role yet are hard to accurately model. To this end, this paper presents an object-aware global-local correspondence architecture, which enables to extract the inter-frame temporal coherent object-level features for accurate VOS. Specifically, we first generate a set of object masks by the ground-truth segmentation, and then we squeeze the current frame representation inside the object masks into a set of global object embeddings. Second, we compute the similarity between each embedding and the feature map, producing an object-aware weight for each pixel. The object-aware feature at each pixel is then constructed by summing the object embeddings weighted by their corresponding object-aware weights, which is able to capture rich object category information. Third, to establish the accurate correspondences between the inter-frame temporal coherent cues, we further design a novel global-local correspondence module to refine the temporal feature representations. Finally, we augment the object-aware features with the global-local aligned information to produce a strong spatio-temporal representation, which is essential to a more reliable pixel-wise segmentation prediction. Extensive evaluations are conducted on three popular VOS benchmarks containing Youtube-VOS, Davis2017 and Davis2016, demonstrating that the proposed method achieves favourable performance compared to the state-of-the-arts.Keywords
Funding Information
- National Key Research and Development Program of China (2018AAA0100400)
- Natural Science Foundation of China (61872189, 61825601)
- 333 High-level Talents Cultivation Project of Jiangsu Province (BRA2020291)
This publication has 55 references indexed in Scilit:
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Semantic Segmentation of Small Objects and Modeling of Uncertainty in Urban Remote Sensing Images Using Deep Convolutional Neural Networks2016
- Deep Residual Learning for Image Recognition2016
- Video Segmentation via Object Flow2016
- A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation2016
- FlowNet: Learning Optical Flow with Convolutional Networks2015
- Coarse-to-Fine Semantic Video Segmentation Using Supervoxel Trees2013
- Feature Correspondence Via Graph Matching: Models and Global Optimization2008
- Learning to detect natural image boundaries using local brightness, color, and texture cuesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2004
- Modal matching for correspondence and recognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 1995