Cops-Ref: A New Dataset and Task on Compositional Referring Expression Comprehension
Authors:
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10083-10092
Abstract
Referring expression comprehension (REF) aims at identifying a particular object in a scene by a natural language expression. It requires joint reasoning over the textual and visual domains to solve the problem. Some popular referring expression datasets, however, fail to provide an ideal test bed for evaluating the reasoning ability of the models, mainly because 1) their expressions typically describe only some simple distinctive properties of the object and 2) their images contain limited distracting information. To bridge the gap, we propose a new dataset for visual reasoning in context of referring expression comprehension with two main features. First, we design a novel expression engine rendering various reasoning logics that can be flexibly combined with rich visual properties to generate expressions with varying compositionality. Second, to better exploit the full reasoning chain embodied in an expression, we propose a new test setting by adding additional distracting images containing objects sharing similar properties with the referent, thus minimising the success rate of reasoning-free cross-domain alignment. We evaluate several state-of-the-art REF models, but find none of them can achieve promising performance. A proposed modular hard mining strategy performs the best but still leaves substantial room for improvement.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Structured Matching for Phrase Localization2016
- Deep Residual Learning for Image Recognition2016
- Natural Language Object Retrieval2016
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015
- Deep visual-semantic alignments for generating image descriptions2015
- What Are You Talking About? Text-to-Image Coreference2014
- ReferItGame: Referring to Objects in Photographs of Natural Scenes2014
- A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their SemanticsInternational Journal of Computer Vision, 2013
- Recognition using visual phrases2011
- WordNetCommunications of the ACM, 1995