Transferable Decoding with Visual Entities for Zero-Shot Image Captioning
Authors:
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3113-3123
Abstract
Image-to-text generation aims to describe images using natural language. Recently, zero-shot image captioning based on pre-trained vision-language models (VLMs) and large language models (LLMs) has made significant progress. However, we have observed and empirically demonstrated that these methods are susceptible to modality bias induced by LLMs and tend to generate descriptions containing objects (entities) that do not actually exist in the image but frequently appear during training (i.e., object hallucination). In this paper, we propose ViECap, a transferable decoding model that leverages entity-aware decoding to generate descriptions in both seen and unseen scenarios. ViECap incorporates entity-aware hard prompts to guide LLMs’ attention toward the visual entities present in the image, enabling coherent caption generation across diverse scenes. With entity-aware hard prompts, ViECap is capable of maintaining performance when transferring from in-domain to out-of-domain scenarios. Extensive experiments demonstrate that ViECap sets a new state-of-the-art cross-domain (transferable) captioning and performs competitively in-domain captioning compared to previous VLMs-based zero-shot methods. Our code is available at: https://github.com/FeiElysia/ViECapKeywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
This publication has 37 references indexed in Scilit:
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- StyleNet: Generating Attractive Visual Captions with Styles2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Deep Residual Learning for Image Recognition2016
- Image Captioning with Semantic Attention2016
- Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data2016
- Mind's eye: A recurrent visual representation for image caption generation2015
- Deep visual-semantic alignments for generating image descriptions2015
- CIDEr: Consensus-based image description evaluation2015
- Microsoft COCO: Common Objects in Context2014