Question-controlled Text-aware Image Captioning
Authors:
Open Access
- 17 October 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3097-3105
Abstract
For an image with multiple scene texts, different people may be interested in different text information. Current text-aware image captioning models are not able to generate distinctive captions according to various information needs. To explore how to generate personalized text-aware captions, we define a new challenging task, namely Question-controlled Text-aware Image Captioning (Qc-TextCap). With questions as control signals, this task requires models to understand questions, find related scene texts and describe them together with objects fluently in human language. Based on two existing text-aware captioning datasets, we automatically construct two datasets, ControlTextCaps and ControlVizWiz to support the task. We propose a novel Geometry and Question Aware Model (GQAM). GQAM first applies a Geometry-informed Visual Encoder to fuse region-level object features and region-level scene text features with considering spatial relationships. Then, we design a Question-guided Encoder to select the most relevant visual features for each question. Finally, GQAM generates a personalized text-aware caption with a Multimodal Decoder. Our model achieves better captioning performance and question answering ability than carefully designed baselines on both two datasets. With questions as control signals, our model generates more informative and diverse captions than the state-of-the-art text-aware captioning model. Our code and datasets are publicly available at https://github.com/HAWLYQ/Qc-TextCap.Keywords
All Related Versions
Funding Information
- Beijing Natural Science Foundation (4192028)
- National Natural Science Foundation of China (61772535)
- National Key R&D Program of China (2020AAA0108600)
This publication has 6 references indexed in Scilit:
- Multimodal Attention with Image Text Spatial Relationship for OCR-Based Image Captioning2020
- Attention on Attention for Image Captioning2019
- Intention Oriented Image Captions With Guiding Objects2019
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Self-Critical Sequence Training for Image Captioning2017
- BLEU2001