Referring Expression Comprehension: A Survey of Methods and Datasets
- 7 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 23 (15209210) , 4426-4440
Abstract
Referring expression comprehension (REC) aims to localize a target object in an image described by a referring expression phrased in natural language. Different from the object detection task that queried object labels have been pre-defined, the REC problem only can observe the queries during the test. It is more challenging than a conventional computer vision problem. This task has attracted a lot of attention from both computer vision and natural language processing community, and several lines of work have been proposed, from CNN-RNN model, modular network to complex graph-based model. In this survey, we first examine the state-of-the-art by comparing modern approaches to the problem. We classify methods by their mechanism to encode the visual and textual modalities. In particular, we examine the common approach of joint embedding images and expressions to a common feature space. We also discuss modular architectures and graph-based models that interface with structured graph representation. In the second part of this survey, we review the datasets available for training and evaluating REC approaches. We then group results according to the datasets, backbone models, settings so that they can be fairly compared. Finally, we discuss promising future directions for this field, in particular the compositional referring expression comprehension that requires more reasoning steps to address.Keywords
All Related Versions
Funding Information
- ARC (DE190100539)
This publication has 59 references indexed in Scilit:
- Modeling Context in Referring Expressions2016
- Grounding of Textual Phrases in Images by Reconstruction2016
- Modeling Context Between Objects for Referring Expression Understanding2016
- LSTM: A Search Space OdysseyIEEE Transactions on Neural Networks and Learning Systems, 2016
- Generation and Comprehension of Unambiguous Object Descriptions2016
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015
- VQA: Visual Question Answering2015
- ReferItGame: Referring to Objects in Photographs of Natural Scenes2014
- A Fast and Accurate Dependency Parser using Neural Networks2014
- The segmented and annotated IAPR TC-12 benchmarkComputer Vision and Image Understanding, 2010