Natural Multimodal Fusion-Based Human–Robot Interaction: Application With Voice and Deictic Posture via Large Language Model
- 5 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics & Automation Magazine
- Vol. 33 (2) , 29-38
Abstract
Translating human intent into robot commands is crucial for the future of service robots in an aging society. Existing human‒robot interaction (HRI) systems relying on gestures or verbal commands are impractical for the elderly, due to difficulties with complex syntax or sign language. To address the challenge, this article introduces a multimodal interaction framework that combines voice and deictic posture information to create a more natural HRI system. Visual cues are first processed by the object detection model to gain a global understanding of the environment, and then bounding boxes are estimated based on depth information. By using a large language model (LLM) with voice-to-text commands and temporally aligned selected bounding boxes, robot action sequences can be generated, while key control syntax constraints are applied to avoid potential LLM hallucination issues. The system is evaluated on real-world tasks with varying levels of complexity, using a Universal Robots UR3e manipulator. Our method demonstrates significantly better HRI performance in terms of accuracy and robustness. To benefit the research community and the general public, we made our code and design open source.Keywords
All Related Versions
Funding Information
- EFRE and MWK ProFö-R&D program (FEIH_ProT_2517820)
- EFRE and MWK ProFö-R&D program (MWK32-7535-30/10/2)
- CAIpirinha—Conversational AI and Personalized Interaction for Risk-Aware Navigation With Human Awareness (BW7_1030/02)
This publication has 17 references indexed in Scilit:
- YOLO-World: Real-Time Open-Vocabulary Object Detection2024
- Communicating human intent to a robotic companion by multi-type gesture sentences2023
- Generating Executable Action Plans with Environmentally-Aware Language Models2023
- Interactive Multimodal Robot Dialog Using Pointing Gesture Recognition2023
- Evaluation of the efficiency of state-of-the-art Speech Recognition enginesProcedia Computer Science, 2022
- Towards Real-Time Physical Human-Robot Interaction Using Skeleton Information and Hand Gestures2018
- Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields2017
- You Only Look Once: Unified, Real-Time Object Detection2016
- An extensible architecture for robust multimodal human-robot communication2013
- Intelligent robotic wheelchair with EMG-, gesture-, and voice-based interfaces2004