Where and Why are They Looking? Jointly Inferring Human Attention and Intentions in Complex Tasks
Authors:
- Ping WeiPWPing Wei
- Yang LiuYLYang Liu
- Tianmin ShuTSTianmin Shu
- Nanning ZhengNZNanning Zheng
- Song-Chun ZhuSZSong-Chun Zhu
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6801-6809
Abstract
This paper addresses a new problem - jointly inferring human attention, intentions, and tasks from videos. Given an RGB-D video where a human performs a task, we answer three questions simultaneously: 1) where the human is looking - attention prediction; 2) why the human is looking there - intention prediction; and 3) what task the human is performing - task recognition. We propose a hierarchical model of human-attention-object (HAO) which represents tasks, intentions, and attention under a unified framework. A task is represented as sequential intentions which transition to each other. An intention is composed of the human pose, attention, and objects. A beam search algorithm is adopted for inference on the HAO graph to output the attention, intention, and task results. We built a new video dataset of tasks, intentions, and attention. It contains 14 task classes, 70 intention categories, 28 object classes, 809 videos, and approximately 330,000 frames. Experiments show that our approach outperforms existing approaches.Keywords
This publication has 30 references indexed in Scilit:
- Augmented saliency model using automatic 3D head pose detection and learned gaze following in natural scenesVision Research, 2015
- Goal-oriented gaze strategies afforded by object interactionVision Research, 2015
- How to Construct a Minimal Theory of MindMind & Language, 2013
- Learning Actionlet Ensemble for 3D Human Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- What/Where to Look Next? Modeling Top-Down Visual Attention in Complex Interactive EnvironmentsIEEE Transactions on Systems, Man, and Cybernetics: Systems, 2013
- From simple innate biases to complex visual conceptsProceedings of the National Academy of Sciences, 2012
- Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search.Psychological Review, 2006
- A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian trackingIEEE Transactions on Signal Processing, 2002
- From the perception of action to the understanding of intentionNature Reviews Neuroscience, 2001
- A model of saliency-based visual attention for rapid scene analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998