Referring Expression Comprehension: A Survey of Methods and Datasets
Preprint
Abstract
Referring expression comprehension (REC) aims to localize a target object in an image described by a referring expression phrased in natural language. Different from the object detection task that queried object labels have been pre-defined, the REC problem only can observe the queries during the test. It thus more challenging than a conventional computer vision problem. This task has attracted a lot of attention from both computer vision and natural language processing community, and several lines of work have been proposed, from CNN-RNN model, modular network to complex graph-based model. In this survey, we first examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to encode the visual and textual modalities. In particular, we examine the common approach of joint embedding images and expressions to a common feature space. We also discuss modular architectures and graph-based models that interface with structured graph representation. In the second part of this survey, we review the datasets available for training and evaluating REC systems. We then group results according to the datasets, backbone models, settings so that they can be fairly compared. Finally, we discuss promising future directions for the field, in particular the compositional referring expression comprehension that requires longer reasoning chain to address.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation2020
- Cops-Ref: A New Dataset and Task on Compositional Referring Expression Comprehension2020
- Multi-Level Policy and Reward-Based Deep Reinforcement Learning Framework for Image CaptioningIEEE Transactions on Multimedia, 2019
- Frame Augmented Alternating Attention Network for Video Question AnsweringIEEE Transactions on Multimedia, 2019
- Deep Hierarchical Encoder–Decoder Network for Image CaptioningIEEE Transactions on Multimedia, 2019
- Know More Say Less: Image Captioning Based on Scene GraphsIEEE Transactions on Multimedia, 2019
- Predicting Visual Features From Text for Image and Video Caption RetrievalIEEE Transactions on Multimedia, 2018
- Visual question answering: A survey of methods and datasetsComputer Vision and Image Understanding, 2017
- GLA: Global–Local Attention for Image DescriptionIEEE Transactions on Multimedia, 2017
- The segmented and annotated IAPR TC-12 benchmarkComputer Vision and Image Understanding, 2010