VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement
Authors:
Preprint
Abstract
In recent years, online Video Instance Segmentation (VIS) methods have shown remarkable advancement with their powerful query-based detectors. Utilizing the output queries of the detector at the frame level, these methods achieve high accuracy on challenging benchmarks. However, we observe the heavy reliance of these methods on the location information that leads to incorrect matching when positional cues are insufficient for resolving ambiguities. Addressing this issue, we present VISAGE that enhances instance association by explicitly leveraging appearance information. Our method involves a generation of queries that embed appearances from backbone feature maps, which in turn get used in our suggested simple tracker for robust associations. Finally, enabling accurate matching in complex scenarios by resolving the issue of over-reliance on location information, we achieve competitive performance on multiple VIS benchmarks. For instance, on YTVIS19 and YTVIS21, our method achieves 54.5 AP and 50.8 AP. Furthermore, to highlight appearance-awareness not fully addressed by existing benchmarks, we generate a synthetic dataset where our method outperforms others significantly by leveraging the appearance cue. Code will be made available at https://github.com/KimHanjung/VISAGE.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- In Defense of Online Models for Video Instance Segmentation2022
- Occluded Video Instance Segmentation: A BenchmarkInternational Journal of Computer Vision, 2022
- Efficient Video Instance Segmentation via Tracklet Query and Proposal2022
- Bridging Composite and Real: Towards End-to-End Deep Image MattingInternational Journal of Computer Vision, 2022
- End-to-End Video Instance Segmentation with Transformers2021
- Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation2021
- STEm-Seg: Spatio-Temporal Embeddings for Instance Segmentation in Videos2020
- End-to-End Object Detection with Transformers2020
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014