Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 11144-11154
Abstract
We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a contrastive image-text pretraining recipe to bridge the gap between image-level pretraining and open-vocabulary object detection. At the pretraining phase, we propose to randomly crop and resize regions of positional embeddings instead of using the whole image positional embeddings. This better matches the use of positional embeddings at region-level in the detection finetuning phase. In addition, we replace the common softmax cross entropy loss in contrastive learning with focal loss to better learn the informative yet difficult examples. Finally, we leverage recent advances in novel object proposals to improve open-vocabulary detection finetuning. We evaluate our full model on the LVIS and COCO open-vocabulary detection benchmarks and zero-shot transfer. RO-ViT achieves a state-of-the-art 32.1 AP r on LVIS, surpassing the best existing approach by +5.8 points in addition to competitive zero-shot transfer detection. Surprisingly, RO-ViT improves the image-level representation as well and achieves the state of the art on 9 out of 12 metrics on COCO and Flickr image-text retrieval benchmarks, outperforming competitive approaches with larger models.Keywords
All Related Versions
This publication has 41 references indexed in Scilit:
- Open-Vocabulary Object Detection Using Captions2021
- VirTex: Learning Visual Representations from Textual Annotations2021
- Seesaw Loss for Long-Tailed Instance Segmentation2021
- Don’t Even Look Once: Synthesizing Features for Zero-Shot Detection2020
- Improved Visual-Semantic Alignment for Zero-Shot Object DetectionProceedings of the AAAI Conference on Artificial Intelligence, 2020
- LVIS: A Dataset for Large Vocabulary Instance Segmentation2019
- Focal Loss for Dense Object Detection2017
- Fine-Grained Image Classification via Combining Vision and Language2017
- Webly Supervised Learning of Convolutional Networks2015
- Learning Models for Object Recognition from Natural Language Descriptions2009