Next‐generation human‐robot interaction with ChatGPT and robot operating system
Authors:
- 26 September 2024
- journal article
- research article
- Published by Wiley in Software: Practice and Experience
- Vol. 55 (2) , 355-382
Abstract
This article presents an innovative concept that harnesses the capabilities of large language models (LLMs) to revolutionize human‐robot interaction. This work aims to connect large language models with the Robot Operating System (ROS), the primary development framework for robotics applications. We develop a package for ROS that seamlessly integrates ChatGPT with ROS2‐based robotic systems. The core idea is to leverage prompt engineering with LLMs, utilizing unique properties such as ability eliciting, chain‐of‐thought, and instruction tuning. The concept employs ontology development to convert unstructured natural language commands into structured robotic instructions specific to the application context through prompt engineering. We capitalize on LLMs' zero‐shots and few‐shots learning capabilities by eliciting structured robotic commands from unstructured human language inputs. To demonstrate the feasibility of this concept, we implemented a proof‐of‐concept that integrates ChatGPT with ROS2, showcasing the transformation of human language instructions into spatial navigation commands for a ROS2‐enabled robot. Besides, we quantitatively evaluated this transformation over three use cases (ground robot, unmanned aerial vehicle, and Robotic arm) and five LLMs (LLaMA‐7b, LLaMA2‐7b, LLaMA2‐70b, GPT‐3.5, and GPT‐4) on a set of 3000 natural language commands. Our system serves as a new stride towards Artificial General Intelligence (AGI) and paves the way for the robotics and natural language processing communities to collaborate in creating novel, intuitive human‐robot interactions. The open‐source implementation of our system on ROS 2 is available on GitHub.Keywords
This publication has 17 references indexed in Scilit:
- End-to-End Speech Recognition: A SurveyIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- ChatGPT for Robotics: A New Approach to Human-Robot Interaction and Task Planning2023
- Improved Trust in Human-Robot Collaboration With ChatGPTIEEE Access, 2023
- SPK-CG: Siamese Network based Posterior Knowledge Selection Model for Knowledge Driven Conversation GenerationACM Transactions on Asian and Low-Resource Language Information Processing, 2023
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- Robot Operating System 2: Design, architecture, and uses in the wildScience Robotics, 2022
- A Survey on End-User Robot ProgrammingACM Computing Surveys, 2021
- GPT-3: Its Nature, Scope, Limits, and ConsequencesMinds and Machines, 2020
- An Ontology for Human-Human Interactions and Learning Interaction Behavior PoliciesACM Transactions on Human-Robot Interaction, 2019
- Robot Operating System (ROS)2017