A Survey of Multimodal Perception Methods for Human–Robot Interaction in Social Environments
Open Access
- 16 October 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Human-Robot Interaction
- Vol. 13 (4) , 1-50
Abstract
Human–robot interaction (HRI) in human social environments (HSEs) poses unique challenges for robot perception systems, which must combine asynchronous, heterogeneous data streams in real time. Multimodal perception systems are well-suited for HRI in HSEs and can provide more rich, robust interaction for robots operating among humans. In this article, we provide an overview of multimodal perception systems being used in HSEs, which is intended to be an introduction to the topic and summary of relevant trends, techniques, resources, challenges, and terminology. We surveyed 15 peer-reviewed robotics and HRI publications over the past 10+ years, providing details about the data acquisition, processing, and fusion techniques used in 65 multimodal perception systems across various HRI domains. Our survey provides information about hardware, software, datasets, and methods currently available for HRI perception research, as well as how these perception systems are being applied in HSEs. Based on the survey, we summarize trends, challenges, and limitations of multimodal human perception systems for robots, then identify resources for researchers and developers and propose future research areas to advance the field.Keywords
This publication has 172 references indexed in Scilit:
- When to Help? A Multimodal Architecture for Recognizing When a User Needs Help from a Social Robot2022
- Multi-modal Data Fusion for People Perception in the Social Robot Haru2022
- Design and Development of a Teleoperation System for Affective Tabletop Robot Haru2021
- Nadine the Social Robot: Three Case Studies in Everyday Life2021
- Iterative Design of an Emotive Voice for the Tabletop Robot Haru2021
- An Exploration of Simple Reactive Responses for Conveying Aliveness Using the Haru Robot2020
- Social Robots for Socio-Physical Distancing2020
- Multimodal and Multi-party Social Interactions2015
- Head Pose Patterns in Multiparty Human-Robot Team-Building InteractionsLecture Notes in Computer Science, 2013
- Perceptual User Interfaces2001