Smallcap: Lightweight Image Captioning Prompted with Retrieval Augmentation
Authors:
Top Cited Papers
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2840-2849
Abstract
Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pretraining and finetuning. As an alternative to large models, we present Smallcap, which generates a caption conditioned on an input image and related captions retrieved from a datastore. Our model is lightweight and fast to train, as the only learned parameters are in newly introduced cross-attention layers between a pre-trained CLIP encoder and GPT-2 decoder. Smallcap can transfer to new domains without additional finetuning and can exploit large-scale data in a training-free fashion since the contents of the datastore can be readily replaced. Our experiments show that Smallcap, trained only on COCO, has competitive performance on this benchmark, and also transfers to other domains without retraining, solely through retrieval from target-domain data. Further improvement is achieved through the training-free exploitation of diverse human-labeled and web data, which proves to be effective for a range of domains, including the nocaps benchmark, designed to test generalization to unseen visual concepts. 1 1 Code: https://github.com/RitaRamo/smallcap.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- A Unified Generation-Retrieval Framework for Image Captioning2019
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning2018
- TGIF: A New Dataset and Benchmark on Animated GIF Description2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- CIDEr: Consensus-based image description evaluation2015
- From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptionsTransactions of the Association for Computational Linguistics, 2014
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- BLEU2001
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989