"Why the face?": Exploring Robot Error Detection Using Instrumented Bystander Reactions
Preprint
Abstract
How do humans recognize and rectify social missteps? We achieve social competence by looking around at our peers, decoding subtle cues from bystanders - a raised eyebrow, a laugh - to evaluate the environment and our actions. Robots, however, struggle to perceive and make use of these nuanced reactions. By employing a novel neck-mounted device that records facial expressions from the chin region, we explore the potential of previously untapped data to capture and interpret human responses to robot error. First, we develop NeckNet-18, a 3D facial reconstruction model to map the reactions captured through the chin camera onto facial points and head motion. We then use these facial responses to develop a robot error detection model which outperforms standard methodologies such as using OpenFace or video data, generalizing well especially for within-participant data. Through this work, we argue for expanding human-in-the-loop robot sensing, fostering more seamless integration of robots into diverse human environments, pushing the boundaries of social cue detection and opening new avenues for adaptable robotics.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- On human-in-the-loop optimization of human–robot interactionNature, 2024
- The Bystander Affect Detection (BAD) Dataset for Failure Detection in HRI2023
- Self-perception of Interaction Errors Through Human Non-verbal Feedback and Robot Context2022
- The building blocks of social competence: Contributions of the Consortium of Individual DevelopmentDevelopmental Cognitive Neuroscience, 2020
- InceptionTime: Finding AlexNet for time series classificationData Mining and Knowledge Discovery, 2020
- A Review on Interactive Reinforcement Learning From Human Social FeedbackIEEE Access, 2020
- Static and Temporal Differences in Social Signals Between Error-Free and Erroneous Situations in Human-Robot Collaboration2019
- OpenFace 2.0: Facial Behavior Analysis Toolkit2018
- LSTM Fully Convolutional Networks for Time Series ClassificationIEEE Access, 2017
- Deep Residual Learning for Image Recognition2016