Filling in the Blank: Rationale-Augmented Prompt Tuning for TextVQA
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1261-1272
Abstract
Recently, generative Text-based visual question answering (TextVQA) methods, which are often based on language models, have exhibited impressive results and drawn increasing attention. However, due to the inconsistencies in both input forms and optimization objectives, the power of pretrained language models is not fully explored, resulting in the need for large amounts of training data. In this work, we rethink the characteristics of the TextVQA task and find that scene text is indeed a special kind of language embedded in images. To this end, we propose a text-centered generative framework FITB (stands for Filling In The Blank), in which multimodal information is mainly represented in textual form and rationale-augmented prompting is involved. Specifically, an infilling-based prompt strategy is utilized to formulate TextVQA as a novel problem of filling in the blank with proper scene text according to the language context. Furthermore, aiming to prevent the model from language bias overfitting, we design a rough answer grounding module to provide visual rationales for promoting multimodal reasoning. Extensive experiments verify the superiority of FITB in both fully-supervised and zero-shot/few-shot settings. Notably, even with a saving of about 64M data, FITB surpasses the state-of-the-art method by 3.00% and 1.99% on TextVQA and ST-VQA datasets, respectively.Keywords
Funding Information
- The Key Research Program of Frontier Sciences, CAS (ZDBS-LY-7024)
This publication has 25 references indexed in Scilit:
- A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations2021
- Spatially Aware Multimodal Transformers for TextVQA2020
- TextCaps: A Dataset for Image Captioning with Reading Comprehension2020
- Cascade Reasoning Network for Text-based Visual Question Answering2020
- Unambiguous Text Localization, Retrieval, and Recognition for Cluttered ScenesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text2020
- Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering2020
- Scene Text Visual Question Answering2019
- From Strings to Things: Knowledge-Enabled VQA Model That Can Read and Reason2019
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016