Cesar Alan Contreras

This paper addresses the problem of planning complex manipulation tasks, in which multiple robots with different end-effectors and capabilities, informed by computer vision, must plan and execute concatenated sequences of actions on a variety of objects that can appear in arbitrary positions and configurations in unstructured scenes. We propose an intent-driven planning pipeline which can robustly construct such action sequences with varying degrees of supervisory input from a human using simple language instructions. The pipeline integrates: (i) perception-to-text scene encoding, (ii) an ensemble of large language models (LLMs) that generate candidate removal sequences based on the operator's intent, (iii) an LLM-based verifier that enforces formatting and precedence constraints, and (iv) a deterministic consistency filter that rejects hallucinated objects. The pipeline is evaluated on an example task in which two robot arms work collaboratively to dismantle an Electric Vehicle battery for recycling applications. A variety of components must be grasped and removed in specific sequences, determined by human instructions and/or by task-order feasibility decisions made by the autonomous system. On 200 real scenes with 600 operator prompts across five component classes, we used metrics of full-sequence correctness and next-task correctness to evaluate and compare five LLM-based planners (including ablation analyses of pipeline components). We also evaluated the LLM-based human interface in terms of time to execution and NASA TLX with human participant experiments. Results indicate that our ensemble-with-verification approach reliably maps operator intent to safe, executable multi-robot plans while maintaining low user effort.

We present a comprehensive teleoperation framework for electric vehicle (EV) battery cell handling, integrating haptic feedback, extended reality (XR) visualization, and task‐parameterized Gaussian mixture regression (TP‐GMR) for adaptive, real‐time trajectory generation. The system enables seamless switching between manual and autonomous operation through a variable autonomy mechanism, while constraint barrier functions (CBFs) enforce spatial safety constraints. A lightweight intent prediction module anticipates user deviation and precomputes corrective trajectories, reducing response time from 2.0 s to under 1 ms. The framework is implemented on an industrial KUKA robotic manipulator and validated in structured and real‐world EV battery disassembly scenarios. Results show that combining XR and haptic feedback reduces task completion time by up to 48% and path deviation by 32%, compared to manual teleoperation without assistance. Predictive replanning improves continuity of force feedback and reduces unnecessary user motion. The integration of XR‐based spatial computing, learning‐from‐demonstration, and real‐time control enables safe, precise, and efficient manipulation in high‐risk environments. This study demonstrates a scalable human‐in‐the‐loop solution for battery recycling and other semi‐structured tasks, where full automation is impractical. The proposed system significantly improves operator performance while maintaining safety and flexibility, marking a meaningful advancement in collaborative field robotics.

Human-robot collaboration requires robots to quickly infer user intent, provide transparent reasoning, and assist users in achieving their goals. Our recent work introduced GUIDER, our framework for inferring navigation and manipulation intents. We propose augmenting GUIDER with a vision-language model (VLM) and a text-only language model (LLM) to form a semantic prior that filters objects and locations based on the mission prompt. A vision pipeline (YOLO for object detection and the Segment Anything Model for instance segmentation) feeds candidate object crops into the VLM, which scores their relevance given an operator prompt; in addition, the list of detected object labels is ranked by a text-only LLM. These scores weight the existing navigation and manipulation layers of GUIDER, selecting context-relevant targets while suppressing unrelated objects. Once the combined belief exceeds a threshold, autonomy changes occur, enabling the robot to navigate to the desired area and retrieve the desired object, while adapting to any changes in the operator's intent. Future work will evaluate the system on Isaac Sim using a Franka Emika arm on a Ridgeback base, with a focus on real-time assistance.

Accurate inference of human intent enables human-robot collaboration without constraining human control or causing conflicts between humans and robots. We present GUIDER (Global User Intent Dual-phase Estimation for Robots), a probabilistic framework that enables a robot to estimate the intent of human operators. GUIDER maintains two coupled belief layers, one tracking navigation goals and the other manipulation goals. In the Navigation phase, a Synergy Map blends controller velocity with an occupancy grid to rank interaction areas. Upon arrival at a goal, an autonomous multi-view scan builds a local 3D cloud. The Manipulation phase combines U2Net saliency, FastSAM instance saliency, and three geometric grasp-feasibility tests, with an end-effector kinematics-aware update rule that evolves object probabilities in real-time. GUIDER can recognize areas and objects of intent without predefined goals. We evaluated GUIDER on 25 trials (five participants x five task variants) in Isaac Sim, and compared it with two baselines, one for navigation and one for manipulation. Across the 25 trials, GUIDER achieved a median stability of 93-100% during navigation, compared with 60-100% for the BOIR baseline, with an improvement of 39.5% in a redirection scenario (T5). During manipulation, stability reached 94-100% (versus 69-100% for Trajectron), with a 31.4% difference in a redirection task (T3). In geometry-constrained trials (manipulation), GUIDER recognized the object intent three times earlier than Trajectron (median remaining time to confident prediction 23.6 s vs 7.8 s). These results validate our dual-phase framework and show improvements in intent inference in both phases of mobile manipulation tasks.

We present a comprehensive teleoperation framework for electric vehicle (EV) battery cell handling, integrating haptic feedback, extended reality (XR) visualisation, and Task-Parameterised Gaussian Mixture Regression (TP-GMR) for adaptive, real-time trajectory generation. The system enables seamless switching between manual and autonomous operation through a variable autonomy mechanism, while Constraint Barrier Functions (CBFs) enforce spatial safety constraints. A lightweight intent prediction module anticipates user deviation and precomputes corrective trajectories, reducing response time from 2.0 seconds to under 1 millisecond. The framework is implemented on an industrial KUKA robotic manipulator and validated in structured and real-world EV battery disassembly scenarios. Results show that combining XR and haptic feedback reduces task completion time by up to 48% and path deviation by 32%, compared to manual teleoperation without assistance. Predictive replanning improves continuity of force feedback and reduces unnecessary user motion. The integration of XR-based spatial computing, learning-from-demonstration, and real-time control enables safe, precise, and efficient manipulation in high-risk environments. This work demonstrates a scalable human-in-the-loop solution for battery recycling and other semi-structured tasks, where full automation is impractical. The proposed system significantly improves operator performance while maintaining safety and flexibility, marking a meaningful advancement in collaborative field robotics.

This paper presents a mini-review of the current state of research in mobile manipulators with variable levels of autonomy, emphasizing their associated challenges and application environments. The need for mobile manipulators in different environments, especially hazardous ones such as decommissioning and search and rescue, is evident due to the unique challenges and risks each presents. Many systems deployed in these environments are not fully autonomous, requiring human-robot teaming to ensure safe and reliable operations under uncertainties. Through this analysis, we identify gaps and challenges in the literature on Variable Autonomy, including cognitive workload and communication delays, and propose future directions, including whole-body Variable Autonomy for mobile manipulators, virtual reality frameworks, and large language models to reduce operators’ complexity and cognitive load in some challenging and uncertain scenarios.

The rapid shift towards electric vehicles (EVs) demands effective end-of-life strategies for lithium-ion batteries (LIBs), necessitating examining recycling methodologies, particularly the disassembly process. This study presents a technoeconomic analysis of EV battery disassembly, focusing on incorporating robotics to address challenges and capitalize on opportunities. Based on the case study of the Mitsubishi Outlander PHEV battery pack, we identify the most labor and cost-intensive components and introduce a structured approach to evaluate automating disassembly tasks. Classifying tasks based on their automation potential, we find 57% of pack-to-module (P2M) tasks readily automatable, with an additional 24% requiring minimal human intervention. Incorporating this into an analysis of task feasibility, level of automation and robot time efficiency, we establish a roadmap towards robotizing the disassembly process based on human-robot colllaboration, demonstrating a 12.85% increase in net annual revenue and 43.75% reduction in disassembly cost with current level of task feasibility. Our findings indicate a long-term roadmap towards increasing level of automation, while surmounting feasibility of complex disassembly tasks remains an unsolved challenge.

This paper presents a mini-review of the current state of research in mobile manipulators with variable levels of autonomy, emphasizing their associated challenges and application environments. The need for mobile manipulators in different environments is evident due to the unique challenges and risks each presents. Many systems deployed in these environments are not fully autonomous, requiring human-robot teaming to ensure safe and reliable operations under uncertainties. Through this analysis, we identify gaps and challenges in the literature on Variable Autonomy, including cognitive workload and communication delays, and propose future directions, including whole-body Variable Autonomy for mobile manipulators, virtual reality frameworks, and large language models to reduce operators' complexity and cognitive load in some challenging and uncertain scenarios.

In this study, we introduce ”SARDiM,” a modular semi-autonomous platform enhanced with mixed reality for industrial disassembly tasks. Through a case study focused on EV battery disassembly, SARDiM integrates Mixed Reality, object segmentation, teleoperation, force feedback, and Variable Autonomy. Utilising the ROS, Unity, and MATLAB platforms, alongside a joint impedance controller, SARDiM facilitates teleoperated disassembly. The approach combines FastSAM for real-time object segmentation, generating data which is subsequently processed through a cluster analysis algorithm to determine the centroid and orientation of the components, categorizing them by size and disassembly priority. This data guides the MoveIt platform in trajectory planning for the Franka Robot arm. SARDiM provides the capability to switch between two teleoperation modes: 1) manual and 2) semi-autonomous with Variable Autonomy. Each was evaluated using four different Interface Methods (IM): 1) direct view, 2) monitor feed, 3) mixed reality with monitor feed, and 4) point cloud mixed reality. Evaluations across the eight IMs demonstrated a 40.61% decrease in joint limit violations using Mode 2. Moreover, Mode 2-IM4 outperformed Mode 1-IM1 by achieving a 2.33%-time reduction while considerably increasing safety, making it optimal for operating in hazardous environments at a safe distance, with the same ease of use as teleoperation with a direct view of the environment.