Low-Shot Video Object Segmentation
- 19 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (7) , 5538-5555
Abstract
Prior research in video object segmentation (VOS) predominantly relies on videos with dense annotations. However, obtaining pixel-level annotations is both costly and time-intensive. In this work, we highlight the potential of effectively training a VOS model using remarkably sparse video annotations—specifically, as few as one or two labeled frames per training video, yet maintaining near equivalent performance levels. We introduce this innovative training methodology as low-shot video object segmentation, abbreviated as low-shot VOS. Central to this method is the generation of reliable pseudo labels for unlabeled frames during the training phase, which are then used in tandem with labeled frames to optimize the model. Notably, our strategy is extremely simple and can be incorporated into the vast majority of current VOS models. For the first time, we propose a universal method for training VOS models on one-shot and two-shot VOS datasets. In the two-shot configuration, utilizing just 7.3% and 2.9% of labeled data from the YouTube-VOS and DAVIS benchmarks respectively, our model delivers results on par with those trained on completely labeled datasets. It is also worth noting that in the one-shot setting, a minor performance decrement is observed in comparison to models trained on fully annotated datasets.Keywords
Funding Information
- Brain-like General Vision Model and Applications project (2022ZD0160402)
- China Postdoctoral Science Foundation (2023M740079, GZC20230058)
- National Key Laboratory of Intelligent Parallel Technology (2024JK15)
This publication has 56 references indexed in Scilit:
- A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model2022
- XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model2022
- Video Object Segmentation with Episodic Graph Memory Networks2020
- Guided Collaborative Training for Pixel-Wise Semi-Supervised Learning2020
- Kernelized Memory Network for Video Object Segmentation2020
- Learning What to Learn for Video Object Segmentation2020
- Collaborative Video Object Segmentation by Foreground-Background Integration2020
- PReMVOS: Proposal-Generation, Refinement and Merging for Video Object Segmentation2019
- Video Object Segmentation with Joint Re-identification and Attention-Aware Mask Propagation2018
- YouTube-VOS: Sequence-to-Sequence Video Object Segmentation2018