Taming Self-Training for Open-Vocabulary Object Detection
Preprint
Abstract
Recent studies have shown promising performance in open-vocabulary object detection (OVD) by utilizing pseudo labels (PLs) from pretrained vision and language models (VLMs). However, teacher-student self-training, a powerful and widely used paradigm to leverage PLs, is rarely explored for OVD. This work identifies two challenges of using self-training in OVD: noisy PLs from VLMs and frequent distribution changes of PLs. To address these challenges, we propose SAS-Det that tames self-training for OVD from two key perspectives. First, we present a split-and-fusion (SAF) head that splits a standard detection into an open-branch and a closed-branch. This design can reduce noisy supervision from pseudo boxes. Moreover, the two branches learn complementary knowledge from different training data, significantly enhancing performance when fused together. Second, in our view, unlike in closed-set tasks, the PL distributions in OVD are solely determined by the teacher model. We introduce a periodic update strategy to decrease the number of updates to the teacher, thereby decreasing the frequency of changes in PL distributions, which stabilizes the training process. Extensive experiments demonstrate SAS-Det is both efficient and effective. SAS-Det outperforms recent models of the same scale by a clear margin and achieves 37.4 AP50 and 29.1 APr on novel categories of the COCO and LVIS benchmarks, respectively.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- Open-Vocabulary DETR with Conditional Matching2022
- PseCo: Pseudo Labeling and Consistency Training for Semi-Supervised Object Detection2022
- Exploring Plain Vision Transformer Backbones for Object Detection2022
- Detecting Twenty-Thousand Classes Using Image-Level Supervision2022
- Single-Stream Multi-level Alignment for Vision-Language Pretraining2022
- X-DETR: A Versatile Architecture for Instance-wise Vision-Language Tasks2022
- End-to-End Object Detection with Transformers2020
- Connecting Vision and Language with Localized Narratives2020
- Zero-Shot Object Detection2018
- Microsoft COCO: Common Objects in Context2014