Wansong Liu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Wansong Liu has a total of 11 co-authors. Here are the top co-authors by number of shared publications:
- Minghui Zheng
Texas A&M University
7 shared publications - Xiao Liang
Texas A&M University
7 shared publications - Kareem Eltouny
Department of Civil, Structural and Environmental Engineering, University at Buffalo, The State University of New York, Buffalo, NY 14260, USA
3 shared publications - Sibo Tian
J. Mike Walker 66 Department of Mechanical Engineering, Texas AM University, College Station, TX, USA
3 shared publications - Chang Liu
Texas A&M University
1 shared publication - Meng-Lun Lee
Department of Mechanical and Aerospace Engineering, University at Buffalo, The State University of New York, Buffalo, New York 14260, USA
1 shared publication
Latest Publications
- 28 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 22 (15455955) , 5061-5071
KG-Planner: Knowledge-Informed Graph Neural Planning for Collaborative Manipulators
This paper presents a novel knowledge-informed graph neural planner (KG-Planner) to address the challenge of efficiently planning collision-free motions for robots in high-dimensional spaces, considering both static and dynamic environments involving humans. Unlike traditional motion planners that struggle with finding a balance between efficiency and optimality, the KG-Planner takes a different approach. Instead of relying solely on a neural network or imitating the motions of an oracle planner, our KG-Planner integrates explicit physical knowledge from the workspace. The integration of knowledge has two key aspects: 1) We present an approach to design a graph that can comprehensively model the workspace’s compositional structure. The designed graph explicitly incorporates critical elements such as robot joints, obstacles, and their interconnections. This representation allows us to capture the intricate relationships between these elements; 2) We train a Graph Neural Network (GNN) that excels at generating nearly optimal robot motions. In particular, the GNN employs a layer-wise propagation rule to facilitate the exchange and update of information among workspace elements based on their connections. This propagation emphasizes the influence of these elements throughout the planning process. To validate the efficacy and efficiency of our KG-Planner, we conduct extensive experiments in both static and dynamic environments. These experiments include scenarios with and without human workers. The results of our approach are compared against existing methods, showcasing the superior performance of the KG-Planner. A short video introduction of this work is available via this https://zh.engr.tamu.edu/wp-content/uploads/sites/310/2024/03/KGPlanner.mp4link. Note to Practitioners —This paper was motivated by the problem of human-robot collaboratively working on remanufacturing processes such as disassembly that require human operators and collaborative robots to work closely with each other. The robots need to plan their trajectories efficiently enough to avoid collision with humans and the trajectories need to be short enough to reduce the cycle time. Traditional motion planners usually struggle with finding a balance between efficiency and optimality, which limits wide applications of collaborative robots in remanufacturing systems that are usually less structured than manufacturing systems. This paper suggests a new planning approach that integrates the workspace’s physical information into a graph and leverages deep learning to obtain safe and near-optimal solutions quickly. Experimental studies and observations demonstrated some advantages of this approach including learning capability, efficiency, and optimality, which makes it a great potential approach to be applied to real remanufacturing processes.
- 28 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 22 (15455955) , 5061-5071
KG-Planner: Knowledge-Informed Graph Neural Planning for Collaborative Manipulators
This paper presents a novel knowledge-informed graph neural planner (KG-Planner) to address the challenge of efficiently planning collision-free motions for robots in high-dimensional spaces, considering both static and dynamic environments involving humans. Unlike traditional motion planners that struggle with finding a balance between efficiency and optimality, the KG-Planner takes a different approach. Instead of relying solely on a neural network or imitating the motions of an oracle planner, our KG-Planner integrates explicit physical knowledge from the workspace. The integration of knowledge has two key aspects: 1) We present an approach to design a graph that can comprehensively model the workspace’s compositional structure. The designed graph explicitly incorporates critical elements such as robot joints, obstacles, and their interconnections. This representation allows us to capture the intricate relationships between these elements; 2) We train a Graph Neural Network (GNN) that excels at generating nearly optimal robot motions. In particular, the GNN employs a layer-wise propagation rule to facilitate the exchange and update of information among workspace elements based on their connections. This propagation emphasizes the influence of these elements throughout the planning process. To validate the efficacy and efficiency of our KG-Planner, we conduct extensive experiments in both static and dynamic environments. These experiments include scenarios with and without human workers. The results of our approach are compared against existing methods, showcasing the superior performance of the KG-Planner. A short video introduction of this work is available via this https://zh.engr.tamu.edu/wp-content/uploads/sites/310/2024/03/KGPlanner.mp4link. Note to Practitioners —This paper was motivated by the problem of human-robot collaboratively working on remanufacturing processes such as disassembly that require human operators and collaborative robots to work closely with each other. The robots need to plan their trajectories efficiently enough to avoid collision with humans and the trajectories need to be short enough to reduce the cycle time. Traditional motion planners usually struggle with finding a balance between efficiency and optimality, which limits wide applications of collaborative robots in remanufacturing systems that are usually less structured than manufacturing systems. This paper suggests a new planning approach that integrates the workspace’s physical information into a graph and leverages deep learning to obtain safe and near-optimal solutions quickly. Experimental studies and observations demonstrated some advantages of this approach including learning capability, efficiency, and optimality, which makes it a great potential approach to be applied to real remanufacturing processes.
- 27 June 2024
- Published by Elsevier in Mechatronics
- Vol. 102
A hybrid task-constrained motion planning for collaborative robots in intelligent remanufacturing
- 27 June 2024
- Published by Elsevier in Mechatronics
- Vol. 102
A hybrid task-constrained motion planning for collaborative robots in intelligent remanufacturing
- 5 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 29 (4) , 3128-3136
Integrating Uncertainty-Aware Human Motion Prediction Into Graph-Based Manipulator Motion Planning
There has been a growing utilization of industrial robots as complementary collaborators for human workers in remanufacturing sites. Such a human–robot collaboration (HRC) aims to assist human workers in improving the flexibility and efficiency of labor-intensive tasks. In this article, we propose a human-aware motion planning framework for HRC to effectively compute collision-free motions for manipulators when conducting collaborative tasks with humans. We employ a neural human motion prediction model to enable proactive planning for manipulators. Particularly, rather than blindly trusting and utilizing predicted human trajectories in the manipulator planning, we quantify uncertainties of the neural prediction model to further ensure human safety. Moreover, we integrate the uncertainty-aware prediction into a graph that captures key workspace elements and illustrates their interconnections. Then, a graph neural network (GNN) is leveraged to operate on the constructed graph. Consequently, robot motion planning considers both the dependencies among all the elements in the workspace and the potential influence of future movements of human workers. We experimentally validate the proposed planning framework using a 6-degree-of-freedom manipulator in a shared workspace where a human is performing disassembling tasks. The results demonstrate the benefits of our approach in terms of improving the smoothness and safety of HRC. A brief video introduction of this work is available as the supplemental materials.
- 5 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 29 (4) , 3128-3136
Integrating Uncertainty-Aware Human Motion Prediction Into Graph-Based Manipulator Motion Planning
There has been a growing utilization of industrial robots as complementary collaborators for human workers in remanufacturing sites. Such a human–robot collaboration (HRC) aims to assist human workers in improving the flexibility and efficiency of labor-intensive tasks. In this article, we propose a human-aware motion planning framework for HRC to effectively compute collision-free motions for manipulators when conducting collaborative tasks with humans. We employ a neural human motion prediction model to enable proactive planning for manipulators. Particularly, rather than blindly trusting and utilizing predicted human trajectories in the manipulator planning, we quantify uncertainties of the neural prediction model to further ensure human safety. Moreover, we integrate the uncertainty-aware prediction into a graph that captures key workspace elements and illustrates their interconnections. Then, a graph neural network (GNN) is leveraged to operate on the constructed graph. Consequently, robot motion planning considers both the dependencies among all the elements in the workspace and the potential influence of future movements of human workers. We experimentally validate the proposed planning framework using a 6-degree-of-freedom manipulator in a shared workspace where a human is performing disassembling tasks. The results demonstrate the benefits of our approach in terms of improving the smoothness and safety of HRC. A brief video introduction of this work is available as the supplemental materials.
- 19 January 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (3) , 2192-2199
DE-TGN: Uncertainty-Aware Human Motion Forecasting Using Deep Ensembles
Ensuring the safety of human workers in a collaborative environment with robots is of utmost importance. Although accurate pose prediction models can help prevent collisions between human workers and robots, they are still susceptible to critical errors. In this study, we propose a novel approach called deep ensembles of temporal graph neural networks (DE-TGN) that not only accurately forecast human motion but also provide a measure of prediction uncertainty. By leveraging deep ensembles and employing stochastic Monte-Carlo dropout sampling, we construct a volumetric field representing a range of potential future human poses based on covariance ellipsoids. To validate our framework, we conducted experiments using three motion capture datasets including Human3.6M, and two human-robot interaction scenarios, achieving state-of-the-art prediction error. Moreover, we discovered that deep ensembles not only enable us to quantify uncertainty but also improve the accuracy of our predictions.
- 19 January 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (3) , 2192-2199
DE-TGN: Uncertainty-Aware Human Motion Forecasting Using Deep Ensembles
Ensuring the safety of human workers in a collaborative environment with robots is of utmost importance. Although accurate pose prediction models can help prevent collisions between human workers and robots, they are still susceptible to critical errors. In this study, we propose a novel approach called deep ensembles of temporal graph neural networks (DE-TGN) that not only accurately forecast human motion but also provide a measure of prediction uncertainty. By leveraging deep ensembles and employing stochastic Monte-Carlo dropout sampling, we construct a volumetric field representing a range of potential future human poses based on covariance ellipsoids. To validate our framework, we conducted experiments using three motion capture datasets including Human3.6M, and two human-robot interaction scenarios, achieving state-of-the-art prediction error. Moreover, we discovered that deep ensembles not only enable us to quantify uncertainty but also improve the accuracy of our predictions.
- 29 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 28 (4) , 2056-2063
Task-Constrained Motion Planning Considering Uncertainty-Informed Human Motion Prediction for Human–Robot Collaborative Disassembly
While the disassembly of high-precision electronic devices is a predominantly labor-intensive process, collaborative robots provide a promising solution through human–robot collaboration. To ensure efficient yet safe collaboration, this article presents a new way to generate task-constrained and collision-free motion for a collaborative robot operating in a dynamic environment involving human movement, which is traditionally challenging due to the high degree of freedom of the corobot and the uncertainty nature of human motion. We first establish a neural human motion prediction model with quantified uncertainty, and then optimize the configuration of the robot online by taking the human motion and uncertainties into consideration. While such rationale is straightforward in nature, our method explicitly quantified the uncertainty of the neural human prediction model to further enhance the collaboration safety, and integrated the quantified uncertainty into the task-satisfied motion planning in real time to efficiently conduct tasks. Extensive experimental tests and comparison studies have been conducted to validate the efficiency and effectiveness of the proposed planning method.
- 29 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 28 (4) , 2056-2063
Task-Constrained Motion Planning Considering Uncertainty-Informed Human Motion Prediction for Human–Robot Collaborative Disassembly
While the disassembly of high-precision electronic devices is a predominantly labor-intensive process, collaborative robots provide a promising solution through human–robot collaboration. To ensure efficient yet safe collaboration, this article presents a new way to generate task-constrained and collision-free motion for a collaborative robot operating in a dynamic environment involving human movement, which is traditionally challenging due to the high degree of freedom of the corobot and the uncertainty nature of human motion. We first establish a neural human motion prediction model with quantified uncertainty, and then optimize the configuration of the robot online by taking the human motion and uncertainties into consideration. While such rationale is straightforward in nature, our method explicitly quantified the uncertainty of the neural human prediction model to further enhance the collaboration safety, and integrated the quantified uncertainty into the task-satisfied motion planning in real time to efficiently conduct tasks. Extensive experimental tests and comparison studies have been conducted to validate the efficiency and effectiveness of the proposed planning method.
- 5 December 2022
- Published by SAGE Publications in Journal of Vibration and Control
- Vol. 29 (19-20) , 4757-4771
Integration of acoustic compliance and noise mitigation in path planning for drones in human–robot collaborative environments
This work presents a framework aimed at mitigating adverse effects of high-amplitude drone noise ranging from hearing loss to reduced productivity in human–robot collaborative environments by infusing acoustic awareness in a path planning algorithm without imposing any additional design layers or hardware to an operational drone. Following a detailed outline of the proposed approach, it is shown that a significant reduction of noise levels perceived by human workers at noise-sensitive locations is realized via a path planner which generates optimal paths ranging from quietest to shortest paths. The approach is then augmented with a path-correction mechanism which accounts for noise exposure duration to ensure the aforementioned optimal paths are compliant with a given industrial/environmental standard. The correction mechanism enforces an adjustment of subsets of the planned paths inside quiet zones designated around noise-sensitive locations. The presented concepts were verified using numerical simulations conducted for a 2-dimensional rasterized obstacle field followed by a statistical design of experiments. The proposed framework is highly versatile and integrable with widely used industrial path planners, rendering it a highly valuable tool for noisy collaborative workplaces.
- 5 December 2022
- Published by SAGE Publications in Journal of Vibration and Control
- Vol. 29 (19-20) , 4757-4771
Integration of acoustic compliance and noise mitigation in path planning for drones in human–robot collaborative environments
This work presents a framework aimed at mitigating adverse effects of high-amplitude drone noise ranging from hearing loss to reduced productivity in human–robot collaborative environments by infusing acoustic awareness in a path planning algorithm without imposing any additional design layers or hardware to an operational drone. Following a detailed outline of the proposed approach, it is shown that a significant reduction of noise levels perceived by human workers at noise-sensitive locations is realized via a path planner which generates optimal paths ranging from quietest to shortest paths. The approach is then augmented with a path-correction mechanism which accounts for noise exposure duration to ensure the aforementioned optimal paths are compliant with a given industrial/environmental standard. The correction mechanism enforces an adjustment of subsets of the planned paths inside quiet zones designated around noise-sensitive locations. The presented concepts were verified using numerical simulations conducted for a 2-dimensional rasterized obstacle field followed by a statistical design of experiments. The proposed framework is highly versatile and integrable with widely used industrial path planners, rendering it a highly valuable tool for noisy collaborative workplaces.
- 7 July 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 53 (1) , 438-450
Robot-Assisted Disassembly Sequence Planning With Real-Time Human Motion Prediction
This article presents a disassembly task planning algorithm considering human–robot collaboration (HRC) and human behavior prediction (HBP). Unlike assembly procedures, the disassembly of end-of-life (EOL) products has been a labor-intensive process with uncertainties difficult to cope with. Meanwhile, it is usually challenging to obtain an optimal sequence efficiently without excessive computational cost. Also, the conventional human-centered task planning, in which the robot has to halt frequently due to unsafe interruptions by human motions, may decrease the efficiency of the disassembly process. In this article, a sequence planner is proposed to assign tasks in real time between a human operator and a robot to overcome the aforementioned challenges. The cost function includes the effort of the human and the robot in terms of both movement distance and time spent on the tasks. The constraints include the disassembly rules and the safety of the human operation. The optimal sequence is generated by solving an optimization problem in a receding-horizon way. In particular, at each step, the proposed disassembly sequence planner locates the workers (a human operator and a robot) and the to-be-disassembled components, predicts human movement for the next several steps, and obtains the optimal disassembly sequence for the next several steps following disassembly rules and safety constraints. Experiments have been extensively conducted on the disassembly of a wooden toybox and a used hard disk drive (HDD) to validate the proposed disassembly sequence planner. The planner has successfully generated the disassembly sequence in an HRC setting explicitly considering real-time human motion prediction and assigned the human operator and the robot to collaboratively complete disassembly tasks without violating disassembly rules and safety constraints.
- 7 July 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Systems, Man, and Cybernetics: Systems
- Vol. 53 (1) , 438-450
Robot-Assisted Disassembly Sequence Planning With Real-Time Human Motion Prediction
This article presents a disassembly task planning algorithm considering human–robot collaboration (HRC) and human behavior prediction (HBP). Unlike assembly procedures, the disassembly of end-of-life (EOL) products has been a labor-intensive process with uncertainties difficult to cope with. Meanwhile, it is usually challenging to obtain an optimal sequence efficiently without excessive computational cost. Also, the conventional human-centered task planning, in which the robot has to halt frequently due to unsafe interruptions by human motions, may decrease the efficiency of the disassembly process. In this article, a sequence planner is proposed to assign tasks in real time between a human operator and a robot to overcome the aforementioned challenges. The cost function includes the effort of the human and the robot in terms of both movement distance and time spent on the tasks. The constraints include the disassembly rules and the safety of the human operation. The optimal sequence is generated by solving an optimization problem in a receding-horizon way. In particular, at each step, the proposed disassembly sequence planner locates the workers (a human operator and a robot) and the to-be-disassembled components, predicts human movement for the next several steps, and obtains the optimal disassembly sequence for the next several steps following disassembly rules and safety constraints. Experiments have been extensively conducted on the disassembly of a wooden toybox and a used hard disk drive (HDD) to validate the proposed disassembly sequence planner. The planner has successfully generated the disassembly sequence in an HRC setting explicitly considering real-time human motion prediction and assigned the human operator and the robot to collaboratively complete disassembly tasks without violating disassembly rules and safety constraints.
- 8 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 27 (6) , 5287-5295
Dynamic Model Informed Human Motion Prediction Based on Unscented Kalman Filter
Human motion prediction is the foundation stone of human–robot collaboration in intelligent manufacturing. The nonlinear and stochastic nature of human motion has made it challenging to predict the motion accurately. Many recent deep-learning-based approaches, e.g., convolutional neural networks or recurrent neural networks (RNNs), have been applied to address this challenge. On the other hand, existing works tend to ignore the importance of human dynamics in motion prediction, especially the effect of muscle force on the motion. This article proposes a novel dynamic model informed motion prediction method. It utilizes an unscented Kalman filter (UKF) to predict the state of the future arm dynamic model such that the future motion of the human arm can be obtained. In particular, the arm dynamic model is developed based on Lagrangian mechanics and represented by differential equations. Embracing the future muscle force predicted by RNN into the differential equations, such a dynamic model is capable of explicitly establishing the intrinsic relation between the future muscle force and the corresponding future arm motion. UKF is leveraged to predict the future joint position and velocity of the human arm based on the dynamic model. Experiments on three motion datasets validate that the proposed prediction method, compared with the traditional RNN-based prediction using skeleton vectors, significantly improves the prediction accuracy regarding elbow and wrist positions.
- 8 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 27 (6) , 5287-5295
Dynamic Model Informed Human Motion Prediction Based on Unscented Kalman Filter
Human motion prediction is the foundation stone of human–robot collaboration in intelligent manufacturing. The nonlinear and stochastic nature of human motion has made it challenging to predict the motion accurately. Many recent deep-learning-based approaches, e.g., convolutional neural networks or recurrent neural networks (RNNs), have been applied to address this challenge. On the other hand, existing works tend to ignore the importance of human dynamics in motion prediction, especially the effect of muscle force on the motion. This article proposes a novel dynamic model informed motion prediction method. It utilizes an unscented Kalman filter (UKF) to predict the state of the future arm dynamic model such that the future motion of the human arm can be obtained. In particular, the arm dynamic model is developed based on Lagrangian mechanics and represented by differential equations. Embracing the future muscle force predicted by RNN into the differential equations, such a dynamic model is capable of explicitly establishing the intrinsic relation between the future muscle force and the corresponding future arm motion. UKF is leveraged to predict the future joint position and velocity of the human arm based on the dynamic model. Experiments on three motion datasets validate that the proposed prediction method, compared with the traditional RNN-based prediction using skeleton vectors, significantly improves the prediction accuracy regarding elbow and wrist positions.