Scaling Up Vision-Language Pretraining for Image Captioning
Authors:
Top Cited Papers
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 17959-17968
Abstract
In recent years, we have witnessed significant performance boost in the image captioning task based on vision-language pre-training (VLP). Scale is believed to be an important factor for this advance. However, most existing work only focuses on pre-training transformers with moderate sizes (e.g., 12 or 24 layers) on roughly 4 million images. In this paper, we present LEMON O, a LargE-scale iMage captiONer, and provide the first empirical study on the scaling behavior of VLP for image captioning. We use the state-of-the-art Vin VL model as our reference model, which consists of an image feature extractor and a transformer model, and scale the transformer both up and down, with model sizes ranging from 13 to 675 million parameters. In terms of data, we conduct experiments with up to 200 million imagetext pairs which are automatically collected from web based on the alt attribute of the image (dubbed as ALT200M 1 1 The dataset is released at https://github.com/xiaoweihu/ALT200M). Extensive analysis helps to characterize the performance trend as the model size and the pre-training data size increase. We also compare different training recipes, especially for training on large-scale noisy data. As a result, LEMON achieves new state of the arts on several major image captioning benchmarks, including COCO Caption, nocaps, and Conceptual Captions. We also show LEMON can generate captions with long-tail vi-sual concepts when used in a zero-shot manner.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- ActBERT: Learning Global-Local Video-Text Representations2020
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-TrainingProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Unified Vision-Language Pre-Training for Image Captioning and VQAProceedings of the AAAI Conference on Artificial Intelligence, 2020
- What Does BERT with Vision Look At?2020
- nocaps: novel object captioning at scale2019
- VideoBERT: A Joint Model for Video and Language Representation Learning2019
- Passive radar using Starlink transmissions: A theoretical study2019
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers2019
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017