Learning to Represent Image and Text with Denotation Graph
Preprint
Abstract
Learning to fuse vision and language information and representing them is an important research problem with many applications. Recent progresses have leveraged the ideas of pre-training (from language modeling) and attention layers in Transformers to learn representation from datasets containing images aligned with linguistic expressions that describe the images. In this paper, we propose learning representations from a set of implied, visually grounded expressions between image and text, automatically mined from those datasets. In particular, we use denotation graphs to represent how specific concepts (such as sentences describing images) can be linked to abstract and generic concepts (such as short phrases) that are also visually grounded. This type of generic-to-specific relations can be discovered using linguistic analysis tools. We propose methods to incorporate such relations into learning representation. We show that state-of-the-art multimodal learning models can be further improved by leveraging automatically harvested structural relations. The representations lead to stronger empirical results on downstream tasks of cross-modal image retrieval, referring expression, and compositional attribute-object recognition. Both our codes and the extracted denotation graphs on the Flickr30K and the COCO datasets are publically available on https://sha-lab.github.io/DG.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Stacked Cross Attention for Image-Text Matching2018
- Cross-Modal and Hierarchical Modeling of Video and Text2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Linking Image and Text with 2-Way Nets2017
- Dual Attention Networks for Multimodal Reasoning and Matching2017
- From Red Wine to Red Tomato: Composition with Context2017
- Aggregated Residual Transformations for Deep Neural Networks2017
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence ModelsInternational Journal of Computer Vision, 2016
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014