Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models
Preprint
Abstract
We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- A Survey on Deep Learning for Multimodal Data FusionNeural Computation, 2020
- Multimodal Machine Learning: A Survey and TaxonomyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Significance Tests Which May be Applied to Samples from Any PopulationsJournal of the Royal Statistical Society Series B: Statistical Methodology, 1937