VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement
Authors:
Preprint
Abstract
In recent years, online Video Instance Segmentation (VIS) methods have shown remarkable advancement with their powerful query-based detectors. Utilizing the output queries of the detector at the frame-level, these methods achieve high accuracy on challenging benchmarks. However, our observations demonstrate that these methods heavily rely on location information, which often causes incorrect associations between objects. This paper presents that a key axis of object matching in trackers is appearance information, which becomes greatly instructive under conditions where positional cues are insufficient for distinguishing their identities. Therefore, we suggest a simple yet powerful extension to object decoders that explicitly extract embeddings from backbone features and drive queries to capture the appearances of objects, which greatly enhances instance association accuracy. Furthermore, recognizing the limitations of existing benchmarks in fully evaluating appearance awareness, we have constructed a synthetic dataset to rigorously validate our method. By effectively resolving the over-reliance on location information, we achieve state-of-the-art results on YouTube-VIS 2019/2021 and Occluded VIS (OVIS). Code is available at https://github.com/KimHanjung/VISAGE.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- In Defense of Online Models for Video Instance Segmentation2022
- Occluded Video Instance Segmentation: A BenchmarkInternational Journal of Computer Vision, 2022
- Masked-attention Mask Transformer for Universal Image Segmentation2022
- Efficient Video Instance Segmentation via Tracklet Query and Proposal2022
- Bridging Composite and Real: Towards End-to-End Deep Image MattingInternational Journal of Computer Vision, 2022
- End-to-End Video Instance Segmentation with Transformers2021
- Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation2021
- STEm-Seg: Spatio-Temporal Embeddings for Instance Segmentation in Videos2020
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014