Large-Language-Model-Aided Assistive Robot for Single-Operator Bimanual Teleoperation: Introduction and Validation of a Flexible Assistance System
- 29 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics & Automation Magazine
- Vol. 33 (1) , 74-84
Abstract
Bimanual teleoperation tasks are highly demanding for human operators, requiring the simultaneous control of two robotic arms while managing complex coordination and cognitive load. Current approaches to this challenge often rely on rigid control schemes or task-specific automations that do not adapt well to dynamic environments or varied operator needs. This article presents a novel large language model (LLM)-aided bimanual teleoperation assistant (BTLA) that helps operators control dual-arm robots through an intuitive voice command interface and variable autonomy. The BTLA system enables a hybrid control paradigm by combining natural language interaction for an assistive robot arm with direct teleoperation of the dominant robotic arm. Our system implements six core manipulation skills with varying autonomy, ranging from direct mirroring to autonomous object manipulation. The BTLA leverages the LLM to interpret natural language commands and select an appropriate assistance mode based on task requirements and operator preferences. Experimental validation on bimanual object manipulation tasks demonstrates that the BTLA system yields a 240.8% increase in success rate over solo teleoperation and a 69.9% increase over dyadic teleoperation, while significantly reducing operator mental workload. In addition, we validate our approach on a physical dual-arm UR3e robot system, achieving a 90% success rate on challenging soft bottle handling and box transportation tasks.Keywords
Funding Information
- Royal Society (IES/R2/232291)
- EPSRC (EP/Y010523/1)
This publication has 19 references indexed in Scilit:
- Reducing Cognitive Load in Teleoperating Swarms of Robots through a Data-Driven Shared Control Approach2024
- Effects of Shared Control on Cognitive Load and Trust in Teleoperated Trajectory TrackingIEEE Robotics and Automation Letters, 2024
- ProgPrompt: Generating Situated Robot Task Plans using Large Language Models2023
- Learning to Assist Bimanual Teleoperation Using Interval Type-2 Polynomial Fuzzy InferenceIEEE Transactions on Cognitive and Developmental Systems, 2023
- A telerobotic system enabling online switching among various architectures and controllersRobotics and Autonomous Systems, 2023
- Online Recognition of Bimanual Coordination Provides Important Context for Movement Data in Bimanual Teleoperated Robots2021
- Dynamic Grasping with Reachability and Motion Awareness2021
- A Bilateral Dual-Arm Teleoperation Robot System with a Unified Control Architecture2021
- Adaptive interval type-2 fuzzy control for multi-legged underwater robot with input saturation and full-state constraintsInternational Journal of Systems Science, 2021
- Two is not always better than one: Effects of teleoperation and haptic coupling2016