SKDF: A Simple Knowledge Distillation Framework for Distilling Open-Vocabulary Knowledge to Open-world Object Detector
Authors:
- Shuailei MaSMShuailei Ma
- Yuefeng WangYWYuefeng Wang
- Ying WeiYWYing Wei
- Jiaqi FanJFJiaqi Fan
- Enming ZhangEZEnming Zhang
- Xinyu SunXSXinyu Sun
- Peihao ChenPCPeihao Chen
Preprint
Abstract
In this paper, we attempt to specialize the VLM model for OWOD tasks by distilling its open-world knowledge into a language-agnostic detector. Surprisingly, we observe that the combination of a simple \textbf{knowledge distillation} approach and the automatic pseudo-labeling mechanism in OWOD can achieve better performance for unknown object detection, even with a small amount of data. Unfortunately, knowledge distillation for unknown objects severely affects the learning of detectors with conventional structures for known objects, leading to catastrophic forgetting. To alleviate these problems, we propose the \textbf{down-weight loss function} for knowledge distillation from vision-language to single vision modality. Meanwhile, we propose the \textbf{cascade decouple decoding structure} that decouples the learning of localization and recognition to reduce the impact of category interactions of known and unknown objects on the localization learning process. Ablation experiments demonstrate that both of them are effective in mitigating the impact of open-world knowledge distillation on the learning of known objects. Additionally, to alleviate the current lack of comprehensive benchmarks for evaluating the ability of the open-world detector to detect unknown objects in the open world, we propose two benchmarks, which we name "\textbf{StandardSet}$\heartsuit$" and "\textbf{IntensiveSet}$\spadesuit$" respectively, based on the complexity of their testing scenarios. Comprehensive experiments performed on OWOD, MS-COCO, and our proposed benchmarks demonstrate the effectiveness of our methods. The code and proposed dataset are available at \url{https://github.com/xiaomabufei/SKDF}.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Revisiting Open World Object DetectionIEEE Transactions on Circuits and Systems for Video Technology, 2023
- CAT: LoCalization and IdentificAtion Cascade Detection Transformer for Open-World Object Detection2023
- Object Detection in 20 Years: A SurveyProceedings of the IEEE, 2023
- Open-World Object Detection via Discriminative Class Prototype Learning2022
- OW-DETR: Open-world Detection Transformer2022
- Decoupled Metric Network for Single-Stage Few-Shot Object DetectionIEEE Transactions on Cybernetics, 2022
- Learning Open-World Object Proposals Without Learning to ClassifyIEEE Robotics and Automation Letters, 2022
- Faster ILOD: Incremental learning for object detectors based on faster RCNNPattern Recognition Letters, 2020
- Microsoft COCO: Common Objects in Context2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009