Multimodal Contrastive Training for Visual Representation Learning
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6991-7000
Abstract
We develop an approach to learning visual representations that embraces multimodal data, driven by a combination of intra- and inter-modal similarity preservation objectives. Unlike existing visual pre-training methods, which solve a proxy prediction task in a single domain, our method exploits intrinsic data properties within each modality and semantic information from cross-modal correlation simultaneously, hence improving the quality of learned visual representations. By including multimodal training in a unified framework with different types of contrastive losses, our method can learn more powerful and generic visual features. We first train our model on COCO and evaluate the learned visual representations on various downstream tasks including image classification, object detection, and instance segmentation. For example, the visual representations pre-trained on COCO by our method achieve state-of-the-art top-1 validation accuracy of 55.3% on ImageNet classification, under the common transfer protocol. We also evaluate our method on the large-scale Stock images dataset and show its effectiveness on multi-label image tagging, and cross-modal retrieval tasks.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Salient Object SubitizingInternational Journal of Computer Vision, 2017
- Learning Deep Structure-Preserving Image-Text Embeddings2016
- Deep Residual Learning for Image Recognition2016
- You Only Look Once: Unified, Real-Time Object Detection2016
- Unsupervised Visual Representation Learning by Context Prediction2015
- Deep visual-semantic alignments for generating image descriptions2015
- Fully convolutional networks for semantic segmentation2015
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Learning Visual Representations using Images with Captions2007