Chat with the Environment: Interactive Multimodal Perception Using Large Language Models
Authors:
Top Cited Papers
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3590-3596
Abstract
Programming robot behavior in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning ability in few-shot robotic planning. However, it remains challenging to ground LLMs in multimodal sensory input and continuous action output, while enabling a robot to interact with its environment and acquire novel information as its policies unfold. We develop a robot interaction scenario with a partially observable state, which necessitates a robot to decide on a range of epistemic actions in order to sample sensory information among multiple modalities, before being able to execute the task correctly. An interactive perception framework is therefore proposed with an LLM as its backbone, whose ability is exploited to instruct epistemic actions and to reason over the resulting multimodal sensations (vision, sound, haptics, proprioception), as well as to plan an entire task execution based on the interactively acquired information. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behavior in a multimodal environment, while multimodal modules with the context of the environmental state help ground the LLMs and extend their processing ability. The project website can be found at https://matcha-model.github.io/.Keywords
All Related Versions
Funding Information
- German Research Foundation (DFG) in the project Crossmodal Learning (TRR-169)
- China Scholarship Council (CSC)
This publication has 11 references indexed in Scilit:
- Knowledge- and ambiguity-aware robot learning from corrective and evaluative feedbackNeural Computing and Applications, 2023
- Impact Makes a Sound and Sound Makes an Impact: Sound Guides Representations and Explorations2022
- Recognizing object surface material from impact sounds for robot manipulation2022
- Uncertainty-Aware Multi-modal Learning via Cross-Modal Random Network Prediction2022
- Sound-Guided Semantic Image Manipulation2022
- Active Information Acquisition under Arbitrary Unknown Disturbances2021
- Robotic Information Gathering using Semantic Language Instructions2021
- Deep Audio-visual Learning: A SurveyInternational Journal of Automation and Computing, 2021
- Inferring Compact Representations for Efficient Natural Language Understanding of Robot Instructions2019
- Cross Modal DisambiguationLecture Notes in Computer Science, 2006