HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
Authors:
Top Cited Papers
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 20213-20224
Abstract
Building an interactive AI assistant that can perceive, reason, and collaborate with humans in the real world has been a long-standing pursuit in the AI community. This work is part of a broader research effort to develop intelligent agents that can interactively guide humans through performing tasks in the physical world. As a first step in this direction, we introduce HoloAssist, a large-scale egocentric human interaction dataset, where two people collaboratively complete physical manipulation tasks. The task performer executes the task while wearing a mixed-reality headset that captures seven synchronized data streams. The task instructor watches the performer’s egocentric video in real time and guides them verbally. By augmenting the data with action and conversational annotations and observing the rich behaviors of various participants, we present key insights into how human assistants correct mistakes, intervene in the task completion procedure, and ground their instructions to the environment. HoloAssist spans 166 hours of data captured by 350 unique instructor-performer pairs. Furthermore, we construct and present benchmarks on mistake detection, intervention type prediction, and hand forecasting, along with detailed analysis. We expect HoloAssist will provide an important resource for building AI assistants that can fluidly collaborate with humans in the real world. Data can be downloaded at https://holoassist.github.io/.Keywords
This publication has 24 references indexed in Scilit:
- EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devices2022
- GIMO: Gaze-Informed Human Motion Prediction in Context2022
- AssistQ: Affordance-Centric Question-Driven Task Completion for Egocentric Assistant2022
- Towards Open Set Video Anomaly Detection2022
- TL;DW? Summarizing Instructional Videos with Task Relevance and Cross-Modal Saliency2022
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100International Journal of Computer Vision, 2021
- A comprehensive review on deep learning-based methods for video anomaly detectionImage and Vision Computing, 2021
- Unsupervised Learning from Narrated Instruction Videos2016
- Online detection of unusual events in videos via dynamic sparse coding2011
- LARRI: A Language-Based Maintenance and Repair Assistant2005