Large Language Models for Multi-Modal Human-Robot Interaction
Authors:
- Chao WangCWChao Wang
- Stephan HaslerSHStephan Hasler
- Daniel TannebergDTDaniel Tanneberg
- Felix OckerFOFelix Ocker
- Frank JoublinFJFrank Joublin
- Antonello CeravolaACAntonello Ceravola
- Joerg DeigmoellerJDJoerg Deigmoeller
- Michael GiengerMGMichael Gienger
Preprint
Abstract
This paper presents an innovative large language model (LLM)-based robotic system for enhancing multi-modal human-robot interaction (HRI). Traditional HRI systems relied on complex designs for intent estimation, reasoning, and behavior generation, which were resource-intensive. In contrast, our system empowers researchers and practitioners to regulate robot behavior through three key aspects: providing high-level linguistic guidance, creating "atomics" for actions and expressions the robot can use, and offering a set of examples. Implemented on a physical robot, it demonstrates proficiency in adapting to multi-modal inputs and determining the appropriate manner of action to assist humans with its arms, following researchers' defined guidelines. Simultaneously, it coordinates the robot's lid, neck, and ear movements with speech output to produce dynamic, multi-modal expressions. This showcases the system's potential to revolutionize HRI by shifting from conventional, manual state-and-flow design methods to an intuitive, guidance-based, and example-driven approach.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Statler: State-Maintaining Language Models for Embodied Reasoning2024
- CoPAL: Corrective Planning of Robot Actions with Large Language Models2024
- ChatGPT for Robotics: Design Principles and Model AbilitiesIEEE Access, 2024
- Large Language Models as Zero-Shot Human Models for Human-Robot Interaction2023
- ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case ApplicationIEEE Access, 2023
- Improved Trust in Human-Robot Collaboration With ChatGPTIEEE Access, 2023
- A Multimodal Emotional Human–Robot Interaction Architecture for Social Robots Engaged in Bidirectional CommunicationIEEE Transactions on Cybernetics, 2020
- Promoting Interactions Between Humans and Robots Using Robotic Emotional BehaviorIEEE Transactions on Cybernetics, 2015
- Development of a generic method to generate upper-body emotional expressions for different social robotsAdvanced Robotics, 2015
- An auto-adaptable algorithm to generate human-like locomotion for different humanoid robots based on motion capture data2010