Scaling Language-Image Pre-Training via Masking
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 23390-23400
Abstract
We present Fast Language-Image Pre-training (FLIP), a simple and more efficient method for training CLIP [52]. Our method randomly masks out and removes a large portion of image patches during training. Masking allows us to learn from more image-text pairs given the same wall-clock time and contrast more samples per iteration with similar memory footprint. It leads to a favorable trade-off between accuracy and training time. In our experiments on 400 million image-text pairs, FLIP improves both accuracy and speed over the no-masking baseline. On a large diversity of downstream tasks, FLIP dominantly outperforms the CLIP counterparts trained on the same data. Facilitated by the speedup, we explore the scaling behavior of increasing the model size, data size, or training length, and report encouraging results and comparisons. We hope that our work will foster future research on scaling vision-language learning.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challenge2018
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Context Encoders: Feature Learning by Inpainting2016
- Rethinking the Inception Architecture for Computer Vision2016
- Deep visual-semantic alignments for generating image descriptions2015
- ImageNet: A large-scale hierarchical image database2009
- Extracting and composing robust features with denoising autoencoders2008
- Dimensionality Reduction by Learning an Invariant Mapping2006