Object Referring in Videos with Language and Human Gaze
Authors:
- Arun Balajee VasudevanAVArun Balajee Vasudevan
- Dengxin DaiDDDengxin Dai
- Luc Van GoolLGLuc Van Gool
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4129-4138
Abstract
We investigate the problem of object referring (OR) i.e. to localize a target object in a visual scene coming with a language description. Humans perceive the world more as continued video snippets than as static images, and describe objects not only by their appearance, but also by their spatio-temporal context and motion features. Humans also gaze at the object when they issue a referring expression. Existing works for OR mostly focus on static images only, which fall short in providing many such cues. This paper addresses OR in videos with language and human gaze. To that end, we present a new video dataset for OR, with 30, 000 objects over 5, 000 stereo video sequences annotated for their descriptions and gaze. We further propose a novel network model for OR in videos, by integrating appearance, motion, gaze, and spatio-temporal context into one network. Experimental results show that our method effectively utilizes motion cues, human gaze, and spatio-temporal context. Our method outperforms previous OR methods. For dataset and code, please refer https://people.ee.ethz.ch/~arunv/ORGaze.html.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Ask Your Neurons: A Neural-Based Approach to Answering Questions about Images2015
- VQA: Visual Question Answering2015
- Long-term recurrent convolutional networks for visual recognition and description2015
- Deep visual-semantic alignments for generating image descriptions2015
- From captions to visual concepts and back2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Glove: Global Vectors for Word Representation2014
- Studying Relationships between Human Gaze, Description, and Computer Vision2013
- Rapid object detection using a boosted cascade of simple features2005
- Long Short-Term MemoryNeural Computation, 1997