SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-World Object Detector
- 19 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (12) , 11738-11752
Abstract
Open World Object Detection (OWOD) is a novel computer vision task with a considerable challenge, bridging the gap between classic object detection (OD) and real-world object detection. In addition to detecting and classifying seen/known objects, OWOD algorithms are expected to localize all potential unseen/unknown objects and incrementally learn them. The large pre-trained vision-language grounding models (VLM, e.g., GLIP) have rich knowledge about the open world, but are limited by text prompts and cannot localize indescribable objects. However, there are many detection scenarios in which pre-defined language descriptions are unavailable during inference. In this paper, we attempt to specialize the VLM model for OWOD tasks by distilling its open-world knowledge into a language-agnostic detector. Surprisingly, we observe that the simple knowledge distillation approach leads to unexpected performance for unknown object detection, even with a small amount of data. Unfortunately, knowledge distillation for unknown objects severely affects the learning of detectors with conventional structures, leading to catastrophic damage to the model’s ability to learn about known objects. To alleviate these problems, we propose the down-weight training strategy for knowledge distillation from vision-language model to single visual modality one. Meanwhile, we propose the cascade decoupled decoders that decouple the learning of localization and recognition to reduce the impact of category interactions of known and unknown objects on the localization learning process. Ablation experiments demonstrate that both of them are effective in mitigating the impact of open-world knowledge distillation on the learning of known objects. Additionally, to alleviate the current lack of comprehensive benchmarks for evaluating the ability of the open-world detector to detect unknown objects in the open world, we refine the benchmark for evaluating the performance of unknown object detection by augmenting annotations for unknown objects which we name“IntensiveSet$\scriptstyle\spadesuit$ ”. Comprehensive experiments performed on OWOD, MS-COCO, and our proposed benchmarks demonstrate the effectiveness of our methods.
Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61871106, 62441231)
- Key R&D projects of Liaoning Province, China (2024JH2/102500015)
- Fundamental Research Funds for the Central Universities of Ministry of Education (N25BSS034)
This publication has 50 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- UC-OWOD: Unknown-Classified Open World Object Detection2022
- Knowledge Distillation: A SurveyInternational Journal of Computer Vision, 2021
- Faster ILOD: Incremental learning for object detectors based on faster RCNNPattern Recognition Letters, 2020
- End-to-End Object Detection with Transformers2020
- Improving Object Detection with Selective Self-supervised Self-training2020
- Zero-Shot Object Detection2018
- Microsoft COCO: Common Objects in Context2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- ImageNet: A large-scale hierarchical image database2009