Zhi Li

Teleoperation enables controlling complex robot systems remotely, providing the ability to impart human expertise from a distance. However, these interfaces can be complicated to use as it is difficult to contextualize information about robot motion in the workspace from the limited camera feedback. Thus, it is required to study the best manner in which assistance can be provided to the operator that reduces interface complexity and effort required for teleoperation. Some techniques that provide assistance to the operator while freeform teleoperating include: (1) perception augmentation, like augmented reality visual cues and additional camera angles, increasing the information available to the operator; (2) action augmentation, like assistive autonomy and control augmentation, optimized to reduce the effort required by the operator while teleoperating. In this article, we investigate: (1) which aspects of dexterous telemanipulation require assistance; (2) the impact of perception and action augmentation in improving teleoperation performance; and (3) what factors impact the usage of assistance and how to tailor these interfaces based on the operators’ needs and characteristics. The findings from this user study and resulting post-study surveys will help identify task-based and user-preferred perception and augmentation features for teleoperation assistance.

This work compared human-robot shared and supervisory control of remote robots for dexterous manipulation, and examined how the reliability of robot autonomy affects human operator performance, workload, and preference for robot assistance. Specifically, we implemented two human-robot collaboration (HRC) paradigms for remote manipulation: (1) shared control, where humans controlled gross manipulation and the robot autonomy controlled precise manipulation actions, and (2) supervisory control, where the robot autonomy controlled both gross and precise manipulation actions but relied on humans to detect and correct errors. We conducted two user studies: one to compare the effectiveness of the two HRC paradigms when assistive autonomy is reliable, and the other to examine the impact of error type and frequency on tasks and human operators in the two HRC paradigms when assistive autonomy is unreliable. Our results show that: (1) the interface with a higher level of reliable autonomy yields significantly better performance, lower workload, and higher user preference but lower engagement, and (2) the frequency and type of the error have significant impacts on the task performance and human workload but only partially affects the operator's preference and usage of autonomy.

Teleoperation enables complex robot platforms to perform tasks beyond the scope of the current state-of-the-art robot autonomy by imparting human intelligence and critical thinking to these operations. For seamless control of robot platforms, it is essential to facilitate optimal situational awareness of the workspace for the operator through active telepresence cameras. However, the control of these active telepresence cameras adds an additional degree of complexity to the task of teleoperation. In this paper we present our results from the user study that investigates: (1) how the teleoperator learns or adapts to performing the tasks via active cameras modeled after camera placements on the TRINA humanoid robot; (2) the perception-action coupling operators implement to control active telepresence cameras, and (3) the camera preferences for performing the tasks. These findings from the human motion analysis and post-study survey will help us determine desired design features for robot teleoperation interfaces and assistive autonomy.

Motion tracking interfaces are intuitive for free-form teleoperation tasks. However, efficient manipulation control can be difficult with such interfaces because of issues like the interference of unintended motions and the limited precision of human motion control. The limitation in control efficiency reduces the operator's performance and increases their workload and frustration during robot teleoperation. To improve the efficiency, we proposed separating controlled degrees of freedom (DoFs) and adjusting the motion scaling ratio of a motion tracking interface. The motion tracking of handheld controllers from a Virtual Reality system was used for the interface. We separated the translation and rotational control into: 1) two controllers held in the dominant and non-dominant hands and 2) hand pose tracking and trackpad inputs of a controller. We scaled the control mapping ratio based on 1) the environmental constraints and 2) the teleoperator's control speed. We further conducted a user study to investigate the effectiveness of the proposed methods in increasing efficiency. Our results show that the separation of position and orientation control into two controllers and the environment-based scaling methods perform better than their alternatives.

Objective: To investigate the effects of human force anticipation, we conducted an experimental load-pushing task with diverse combinations of informed and actual loading weights. Background: Human motor control tends to rely upon the anticipated workload to plan the force to exert, particularly in fast tasks such as pushing objects in less than 1 s. The motion and force responses in such tasks may depend on the anticipated resistive forces, based on a learning process. Method: Pushing performances of 135 trials were obtained from 9 participants. We varied the workload by changing the masses from 0.2 to 5 kg. To influence anticipation, participants were shown a display of the workload that was either correct or incorrect. We collected the motion and force data, as well as electromyography (EMG) signals from the actively used muscle groups. Results: Overanticipation produced overshoot performances in more than 80% of trials. Lighter actual workloads were also associated with overshoot. Pushing behaviors with heavier workloads could be classified into feedforward-dominant and feedback-dominant responses based on the timing of force, motion, and EMG responses. In addition, we found that the preceding trial condition affected the performance of the subsequent trial. Conclusion: Our results show that the first peak of the pushing force increases consistently with anticipatory workload. Application: This study improves our understanding of human motion control and can be applied to situations such as simulating interactions between drivers and assistive systems in intelligent vehicles.

It is important that automotive engineers understand the interactions between active human maneuvering motions and vehicle dynamics, and how vehicle control affects the physical sensations of the human driver. This paper proposes a new system framework, the human model-based active driving system (HuMADS) for simulating human driver-vehicle interactions. HuMADS integrates the vehicle controller with models of vehicle dynamics and human biomechanics. It has an hierarchical closed-loop architecture for driver-vehicle control systems, including structures and contact interfaces of human and vehicle bodies. HuMADS is based on the OpenSim simulation platform. The developed system regulates the human model dynamics, such that the human model can react realistically to vehicle maneuver motions. The usability of the HuMADS is demonstrated through the simulation of coordinated gas/brake pedal operation and wheel-steering in highway driving tasks. The simulated vehicle dynamics and vehicle maneuvers are comparable with previously published experimental data of car-following driving. In addition, the proposed controllers successfully maintain the human body's balance inside the vehicle during vehicle maneuvers. We are convinced that the HuMADS has potential as a tool for the development of intelligent transportation systems and investigation of integrated safety.

Reach-to-grasp arm postures differ from those in pure reaching because they are affected by grasp position/orientation, rather than simple transport to a position during a reaching motion. This paper investigates this difference via an analysis of experimental data collected on reaching and reach-to-grasp motions. A seven-degree-of-freedom (DOFs) kinematic arm model with the swivel angle is used for the motion analysis. Compared to a widely used anatomical arm model, this model distinguishes clearly the four grasping-relevant DOFs (GR-DOFs) that are affected by positions and orientations of the objects to be grasped. These four GR-DOFs include the swivel angle that measures the elbow rotation about the shoulder–wrist axis, and three wrist joint angles. For each GR-DOF, we quantify position vs orientation task-relevance bias that measures how much the DOF is affected by the grasping position vs orientation. The swivel angle and forearm supination have similar bias, and the analysis of their motion suggests two hypotheses regarding the synergistic coordination of the macro- and micro-structures of the human arm (1) DOFs with similar task-relevance are synergistically coordinated; and (2) such synergy breaks when a task-relevant DOF is close to its joint limit without necessarily reaching the limit. This study provides a motion analysis method to reduce the control complexity for reach-to-grasp tasks, and suggests using dynamic coupling to coordinate the hand and arm of upper-limb exoskeletons.