Multimodal Data Augmentation for Image Captioning using Diffusion Models
Open Access
- 29 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 23-33
Abstract
Image captioning, an important vision-language task, often requires a tremendous number of finely labeled image-caption pairs for learning the underlying alignment between images and texts. In this paper, we proposed a multimodal data augmentation method, leveraging a recent text-to-image model called Stable Diffusion, to expand the training set via high-quality generation of image-caption pairs. Extensive experiments on the MS COCO dataset demonstrate the advantages of our approach over several benchmark methods, and particularly a significant boost when having fewer training instances. In addition, models trained on our augmented datasets also outperform prior unpaired image captioning methods by a large margin. Finally, further improvement regarding the training efficiency and effectiveness can be obtained after intentionally filtering the generated data based on quality assessment.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Image Captioning with Deep Bidirectional LSTMs and Multi-Task LearningACM Transactions on Multimedia Computing, Communications, and Applications, 2018
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Knowing When to Look: Adaptive Attention via a Visual Sentinel for Image Captioning2017
- Self-Critical Sequence Training for Image Captioning2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data2016
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- Microsoft COCO: Common Objects in Context2014