Learning to Detect Novel Species with SAM in the Wild
Open Access
- 13 November 2024
- journal article
- research article
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 133 (5) , 2247-2258
Abstract
This paper tackles the limitation of a closed-world object detection model that was trained on one species. The expectation for this model is that it will not generalize well to recognize the instances of new species if they were present in the incoming data stream. We propose a novel object detection framework for this open-world setting that is suitable for applications that monitor wildlife, ocean life, livestock, plant phenotype and crops that typically feature one species in the image. Our method leverages labeled samples from one species in combination with a novelty detection method and Segment Anything Model, a vision foundation model, to (1) identify the presence of new species in unlabeled images, (2) localize their instances, and (3) retrain the initial model with the localized novel class instances. The resulting integrated system assimilates and learns from unlabeled samples of the new classes while not “forgetting” the original species the model was trained on. We demonstrate our findings on two different domains, (1) wildlife detection and (2) plant detection. Our method achieves an AP of 56.2 (for 4 novel species) to 61.6 (for 1 novel species) for wildlife domain, without relying on any ground truth data in the background.Keywords
This publication has 27 references indexed in Scilit:
- WilDect-YOLO: An efficient and robust computer vision-based accurate object localization model for automated endangered wildlife detectionEcological Informatics, 2022
- Open-Set Semi-Supervised Object Detection2022
- Perspectives in machine learning for wildlife conservationNature Communications, 2022
- Object detection in agricultural contexts: A multiple resolution benchmark and comparison to humanComputers and Electronics in Agriculture, 2021
- A novel method for detecting morphologically similar crops and weeds based on the combination of contour masks and filtered Local Binary Pattern operatorsGigaScience, 2020
- Transductive Semi-Supervised Deep Learning Using Min-Max Features2018
- Smooth Neighbors on Teacher Graphs for Semi-Supervised Learning2018
- Image Segmentation for Fruit Detection and Yield Estimation in Apple OrchardsJournal of Field Robotics, 2017
- Microsoft COCO: Common Objects in Context2014
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009