XBG: End-to-End Imitation Learning for Autonomous Behaviour in Human-Robot Interaction and Collaboration
Authors:
- Carlos Cardenas-PerezCCCarlos Cardenas-Perez
- Giulio RomualdiGRGiulio Romualdi
- Mohamed ElobaidMEMohamed Elobaid
- Stefano DafarraSDStefano Dafarra
- Giuseppe L'ERarioGLGiuseppe L'ERario
- Silvio TraversaroSTSilvio Traversaro
- Pietro MorerioPMPietro Morerio
- Alessio Del BueABAlessio Del Bue
- Daniele PucciDaniele Pucci
- 11 November 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (12) , 11617-11624
Abstract
This paper presents XBG (eXteroceptive Behaviour Generation), a multimodal end-to-end Imitation Learning (IL) system for whole-body autonomous humanoid robots used in real-world Human-Robot Interaction (HRI) scenarios. The main contribution is an architecture for learning HRI behaviours using a data-driven approach. A diverse dataset is collected via teleoperation, covering multiple HRI scenarios, such as handshaking, handwaving, payload reception, walking, and walking with a payload. After synchronizing, filtering, and transforming the data, we show how to train the presented Deep Neural Networks (DNN), integrating exteroceptive and proprioceptive information to help the robot understand both its environment and its actions. The robot takes in sequences of images (RGB and depth) and joints state information to react accordingly. By fusing multimodal signals over time, the model enables autonomous capabilities in a robotic platform. The models are evaluated based on the success rates in the mentioned HRI scenarios and they are deployed on the ergoCub humanoid robot. XBG achieves success rates between 60% and 100% even when tested in unseen environments.Keywords
All Related Versions
Funding Information
- Italian National Institute for Insurance against Accidents at Work
This publication has 11 references indexed in Scilit:
- iCub3 avatar system: Enabling remote fully immersive embodiment of humanoid robotsScience Robotics, 2024
- Deep Imitation Learning for Humanoid Loco-manipulation Through Human Teleoperation2023
- Teleoperation of Humanoid Robots: A SurveyIEEE Transactions on Robotics, 2023
- A vision-based virtual fixture with robot learning for teleoperationRobotics and Autonomous Systems, 2023
- Robotic Vision for Human-Robot Interaction and Collaboration: A Survey and Systematic ReviewACM Transactions on Human-Robot Interaction, 2023
- Visually-guided motion planning for autonomous driving from interactive demonstrationsEngineering Applications of Artificial Intelligence, 2022
- Human–robot collaboration and machine learning: A systematic review of recent researchRobotics and Computer-Integrated Manufacturing, 2022
- Multimodal End-to-End Autonomous DrivingIEEE Transactions on Intelligent Transportation Systems, 2020
- A Benchmarking of DCM-Based Architectures for Position, Velocity and Torque-Controlled Humanoid RobotsInternational Journal of Humanoid Robotics, 2020
- Whole-Body Geometric Retargeting for Humanoid Robots2019