An End-to-End Approach to Natural Language Object Retrieval via Context-Aware Deep Reinforcement Learning
Preprint
Abstract
We propose an end-to-end approach to the natural language object retrieval task, which localizes an object within an image according to a natural language description, i.e., referring expression. Previous works divide this problem into two independent stages: first, compute region proposals from the image without the exploration of the language description; second, score the object proposals with regard to the referring expression and choose the top-ranked proposals. The object proposals are generated independently from the referring expression, which makes the proposal generation redundant and even irrelevant to the referred object. In this work, we train an agent with deep reinforcement learning, which learns to move and reshape a bounding box to localize the object according to the referring expression. We incorporate both the spatial and temporal context information into the training procedure. By simultaneously exploiting local visual information, the spatial and temporal context and the referring language a priori, the agent selects an appropriate action to take at each time. A special action is defined to indicate when the agent finds the referred object, and terminate the procedure. We evaluate our model on various datasets, and our algorithm significantly outperforms the compared algorithms. Notably, the accuracy improvement of our method over the recent method GroundeR and SCRC on the ReferItGame dataset are 7.67% and 18.25%, respectively.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Modeling Context in Referring Expressions2016
- Grounding of Textual Phrases in Images by Reconstruction2016
- Modeling Context Between Objects for Referring Expression Understanding2016
- End-to-End Learning of Action Detection from Frame Glimpses in Videos2016
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Going deeper with convolutions2015
- Show and tell: A neural image caption generator2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Microsoft COCO: Common Objects in Context2014