D-RMGPT: Robot-assisted collaborative tasks driven by large multimodal models
Preprint
Abstract
Collaborative robots are increasingly popular for assisting humans at work and daily tasks. However, designing and setting up interfaces for human-robot collaboration is challenging, requiring the integration of multiple components, from perception and robot task control to the hardware itself. Frequently, this leads to highly customized solutions that rely on large amounts of costly training data, diverging from the ideal of flexible and general interfaces that empower robots to perceive and adapt to unstructured environments where they can naturally collaborate with humans. To overcome these challenges, this paper presents the Detection-Robot Management GPT (D-RMGPT), a robot-assisted assembly planner based on Large Multimodal Models (LMM). This system can assist inexperienced operators in assembly tasks without requiring any markers or previous training. D-RMGPT is composed of DetGPT-V and R-ManGPT. DetGPT-V, based on GPT-4V(vision), perceives the surrounding environment through one-shot analysis of prompted images of the current assembly stage and the list of components to be assembled. It identifies which components have already been assembled by analysing their features and assembly requirements. R-ManGPT, based on GPT-4, plans the next component to be assembled and generates the robot's discrete actions to deliver it to the human co-worker. Experimental tests on assembling a toy aircraft demonstrated that D-RMGPT is flexible and intuitive to use, achieving an assembly success rate of 83% while reducing the assembly time for inexperienced operators by 33% compared to the manual process. http://robotics-and-ai.github.io/LMMmodels/Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection2024
- Improved Baselines with Visual Instruction Tuning2024
- Research on Real-time Detection of Stacked Objects Based on Deep LearningJournal of Intelligent & Robotic Systems, 2023
- Task and Motion Planning with Large Language Models for Object Rearrangement2023
- Large Language Models as Zero-Shot Human Models for Human-Robot Interaction2023
- Chat with the Environment: Interactive Multimodal Perception Using Large Language Models2023
- Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers2023
- PartSLIP: Low-Shot Part Segmentation for 3D Point Clouds via Pretrained Image-Language Models2023
- Symbiotic human-robot collaborative assemblyCIRP Annals, 2019
- Deep learning for smart manufacturing: Methods and applicationsJournal of Manufacturing Systems, 2018