Hoi-Yin Lee
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Hoi-Yin Lee has a total of 26 co-authors. Here are the top co-authors by number of shared publications:
- David Navarro-Alarcon
The Hong Kong Polytechnic University
9 shared publications - Anqing Duan
Robotics Department, Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates
7 shared publications - Peng Zhou
School of Advanced Engineering, Great Bay University, Dongguan, China
6 shared publications - Chenguang Yang
Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China
4 shared publications - Pai Zheng
Department of Industrial and Systems Engineering The Hong Kong Polytechnic University Hong Kong China
3 shared publications - Liuming Qiu
Department of Mechanical Engineering, The Hong Kong Polytechnic University (PolyU), Kowloon, Hong Kong
2 shared publications
Latest Publications
- Open Access
- 7 January 2026
- Published by Elsevier in Robotics and Computer-Integrated Manufacturing
- Vol. 100
Non-prehensile tool-object manipulation by integrating LLM-based planning and manoeuvrability-driven controls
- 7 January 2026
- Published by Elsevier in Robotics and Computer-Integrated Manufacturing
- Vol. 100
Open AccessNon-prehensile tool-object manipulation by integrating LLM-based planning and manoeuvrability-driven controls
- 22 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (7) , 7102-7109
Iterative Shaping of Multi-Particle Aggregates Based on Action Trees and VLM
In this letter, we address the problem of manipulating multi-particle aggregates using a bimanual robotic system. Our approach enables the autonomous transport of dispersed particles through a series of shaping and pushing actions using robotically controlled tools. Achieving this advanced manipulation capability presents two key challenges: high-level task planning and trajectory execution. For task planning, we leverage Vision Language Models (VLMs) to enable primitive actions such as tool affordance grasping and non-prehensile particle pushing. For trajectory execution, we represent the evolving particle aggregate's contour using truncated Fourier series, providing efficient parametrization of its closed shape. We adaptively compute trajectory waypoints based on group cohesion and the geometric centroid of the aggregate, accounting for its spatial distribution and collective motion. Through real-world experiments, we demonstrate the effectiveness of our methodology in actively shaping and manipulating multi-particle aggregates while maintaining high system cohesion.
- 22 May 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (7) , 7102-7109
Iterative Shaping of Multi-Particle Aggregates Based on Action Trees and VLM
In this letter, we address the problem of manipulating multi-particle aggregates using a bimanual robotic system. Our approach enables the autonomous transport of dispersed particles through a series of shaping and pushing actions using robotically controlled tools. Achieving this advanced manipulation capability presents two key challenges: high-level task planning and trajectory execution. For task planning, we leverage Vision Language Models (VLMs) to enable primitive actions such as tool affordance grasping and non-prehensile particle pushing. For trajectory execution, we represent the evolving particle aggregate's contour using truncated Fourier series, providing efficient parametrization of its closed shape. We adaptively compute trajectory waypoints based on group cohesion and the geometric centroid of the aggregate, accounting for its spatial distribution and collective motion. Through real-world experiments, we demonstrate the effectiveness of our methodology in actively shaping and manipulating multi-particle aggregates while maintaining high system cohesion.
- 18 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74 (00189456) , 1-12
PSO-Based Optimal Coverage Path Planning for Surface Defect Inspection of 3C Components With a Robotic Line Scanner
The automatic inspection of surface defects is an essential task for quality control in the computers, communications, and consumer electronics (3C) industry. Traditional inspection mechanisms (viz. line-scan sensors) have a limited field of view, thus, prompting the necessity for a multifaceted robotic inspection system capable of comprehensive scanning. Optimally selecting the robot’s viewpoints and planning a path is regarded as coverage path planning (CPP), a problem that enables inspecting the object’s complete surface while reducing the scanning time and avoiding misdetection of defects. In this paper, we present a new approach for robotic line scanners to detect surface defects of 3C free-form objects automatically. A two-stage region segmentation method defines the local scanning based on the random sample consensus (RANSAC) and K-means clustering to improve the inspection coverage. The proposed method also consists of an adaptive region-of-interest (ROI) algorithm to define the local scanning paths. Besides, a Particle Swarm Optimization (PSO)-based method is used for global inspection path generation to minimize the inspection time. The developed method is validated by simulation-based and experimental studies on various free-form workpieces, and its performance is compared with that of two state-of-the-art solutions. The reported results demonstrate the feasibility and effectiveness of our proposed method.
- 18 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 74 (00189456) , 1-12
PSO-Based Optimal Coverage Path Planning for Surface Defect Inspection of 3C Components With a Robotic Line Scanner
The automatic inspection of surface defects is an essential task for quality control in the computers, communications, and consumer electronics (3C) industry. Traditional inspection mechanisms (viz. line-scan sensors) have a limited field of view, thus, prompting the necessity for a multifaceted robotic inspection system capable of comprehensive scanning. Optimally selecting the robot’s viewpoints and planning a path is regarded as coverage path planning (CPP), a problem that enables inspecting the object’s complete surface while reducing the scanning time and avoiding misdetection of defects. In this paper, we present a new approach for robotic line scanners to detect surface defects of 3C free-form objects automatically. A two-stage region segmentation method defines the local scanning based on the random sample consensus (RANSAC) and K-means clustering to improve the inspection coverage. The proposed method also consists of an adaptive region-of-interest (ROI) algorithm to define the local scanning paths. Besides, a Particle Swarm Optimization (PSO)-based method is used for global inspection path generation to minimize the inspection time. The developed method is validated by simulation-based and experimental studies on various free-form workpieces, and its performance is compared with that of two state-of-the-art solutions. The reported results demonstrate the feasibility and effectiveness of our proposed method.
- 15 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 21 (4) , 2898-2907
Human-Aware Reactive Task Planning of Sequential Robotic Manipulation Tasks
The recent emergence of Industry 5.0 underscores the need for increased autonomy in human–robot interaction (HRI), presenting both motivation and challenges in achieving resilient and energy-efficient production systems. To address this, in this article, we introduce a strategy for seamless collaboration between humans and robots in manufacturing and maintenance tasks. Our method enables smooth switching between temporary HRI (human-aware mode) and long-horizon automated manufacturing (fully automatic mode), effectively solving the human–robot coexistence problem. We develop a task progress monitor that decomposes complex tasks into robot-centric action sequences, further divided into three-phase subtasks. A trigger signal orchestrates mode switches based on detected human actions and their contribution to the task. In addition, we introduce a human agent coefficient matrix, computed using selected environmental features, to determine cut-points for reactive execution by each robot. To validate our approach, we conducted extensive experiments involving robotic manipulators performing representative manufacturing tasks in collaboration with humans. The results show promise for advancing HRI, offering pathways to enhancing sustainability within Industry 5.0. Our work lays the foundation for intelligent manufacturing processes in future societies, marking a pivotal step toward realizing the full potential of human–robot collaboration.
- 15 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 21 (4) , 2898-2907
Human-Aware Reactive Task Planning of Sequential Robotic Manipulation Tasks
The recent emergence of Industry 5.0 underscores the need for increased autonomy in human–robot interaction (HRI), presenting both motivation and challenges in achieving resilient and energy-efficient production systems. To address this, in this article, we introduce a strategy for seamless collaboration between humans and robots in manufacturing and maintenance tasks. Our method enables smooth switching between temporary HRI (human-aware mode) and long-horizon automated manufacturing (fully automatic mode), effectively solving the human–robot coexistence problem. We develop a task progress monitor that decomposes complex tasks into robot-centric action sequences, further divided into three-phase subtasks. A trigger signal orchestrates mode switches based on detected human actions and their contribution to the task. In addition, we introduce a human agent coefficient matrix, computed using selected environmental features, to determine cut-points for reactive execution by each robot. To validate our approach, we conducted extensive experiments involving robotic manipulators performing representative manufacturing tasks in collaboration with humans. The results show promise for advancing HRI, offering pathways to enhancing sustainability within Industry 5.0. Our work lays the foundation for intelligent manufacturing processes in future societies, marking a pivotal step toward realizing the full potential of human–robot collaboration.
- 19 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 30 (5) , 3254-3265
Bimanual Deformable Bag Manipulation Using a Structure-of-Interest Based Neural Dynamics Model
The manipulation of deformable objects by robotic systems presents significant challenges due to their complex dynamics and infinite-dimensional configuration spaces. This article introduces a novel approach to deformable object manipulation (DOM) by emphasizing the introduction and manipulation of structures of interest (SOIs) in deformable fabric bags. We propose a bimanual manipulation framework that leverages a graph neural network-based neural dynamics model to succinctly represent and predict the behavior of these SOIs. Our approach involves global particle sampling process to construct a particle representation from partial point clouds of the SOIs and learning the neural dynamics model that effectively captures the essential deformations of the SOIs for fabric bags. By integrating this neural dynamics model with model predictive control, we enable robotic manipulators to perform precise and stable manipulation tasks focused on the SOIs. We validate our new framework through various experiments that demonstrate its efficacy in manipulating deformable bags and T-shirts. Our contributions not only address the complexities inherent in DOM, but also provide new perspectives and methodologies for enhancing robotic interactions with deformable materials by concentrating on their critical structural elements.
- 19 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 30 (5) , 3254-3265
Bimanual Deformable Bag Manipulation Using a Structure-of-Interest Based Neural Dynamics Model
The manipulation of deformable objects by robotic systems presents significant challenges due to their complex dynamics and infinite-dimensional configuration spaces. This article introduces a novel approach to deformable object manipulation (DOM) by emphasizing the introduction and manipulation of structures of interest (SOIs) in deformable fabric bags. We propose a bimanual manipulation framework that leverages a graph neural network-based neural dynamics model to succinctly represent and predict the behavior of these SOIs. Our approach involves global particle sampling process to construct a particle representation from partial point clouds of the SOIs and learning the neural dynamics model that effectively captures the essential deformations of the SOIs for fabric bags. By integrating this neural dynamics model with model predictive control, we enable robotic manipulators to perform precise and stable manipulation tasks focused on the SOIs. We validate our new framework through various experiments that demonstrate its efficacy in manipulating deformable bags and T-shirts. Our contributions not only address the complexities inherent in DOM, but also provide new perspectives and methodologies for enhancing robotic interactions with deformable materials by concentrating on their critical structural elements.
- 27 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 29 (5) , 3924-3935
Robust Grasping by Bimanual Robots With Stable Parameterization-Based Contact Servoing
Robots with bimanual morphology usually possess higher flexibility, dexterity, and efficiency than those only equipped with a single arm. The dual-arm structure has enabled robots to perform various intricate tasks that are difficult or even impossible to achieve by unimanipulation. In this article, we aim to achieve robust bimanual grasping for object transportation. In particular, provided that stable contact is the key to the success of the transportation task, our focus lies on stabilizing the contact between the object and the robot end-effectors by employing the contact servoing strategy. To ensure that the contact is stable, the contact wrenches are required to evolve within the so-called friction cones all the time throughout the transportation task. To this end, we propose stabilizing the contact by leveraging a novel contact parameterization model. Parameterization expresses the contact stability manifold with a set of constraint-free exogenous parameters where the mapping is bijective. Notably, such parameterization can guarantee that the contact stability constraints can always be satisfied. We also show that many commonly used contact models can be parameterized out of a similar principle. Furthermore, to exploit the parameterized contact models in the control law, we devise a contact servoing strategy for the bimanual robotic system such that the force feedback signals from the force/torque sensors are incorporated into the control loop. The effectiveness of the proposed approach is well demonstrated with the experiments on several representative bimanual transportation tasks.
- 27 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ASME Transactions on Mechatronics
- Vol. 29 (5) , 3924-3935
Robust Grasping by Bimanual Robots With Stable Parameterization-Based Contact Servoing
Robots with bimanual morphology usually possess higher flexibility, dexterity, and efficiency than those only equipped with a single arm. The dual-arm structure has enabled robots to perform various intricate tasks that are difficult or even impossible to achieve by unimanipulation. In this article, we aim to achieve robust bimanual grasping for object transportation. In particular, provided that stable contact is the key to the success of the transportation task, our focus lies on stabilizing the contact between the object and the robot end-effectors by employing the contact servoing strategy. To ensure that the contact is stable, the contact wrenches are required to evolve within the so-called friction cones all the time throughout the transportation task. To this end, we propose stabilizing the contact by leveraging a novel contact parameterization model. Parameterization expresses the contact stability manifold with a set of constraint-free exogenous parameters where the mapping is bijective. Notably, such parameterization can guarantee that the contact stability constraints can always be satisfied. We also show that many commonly used contact models can be parameterized out of a similar principle. Furthermore, to exploit the parameterized contact models in the control law, we devise a contact servoing strategy for the bimanual robotic system such that the force feedback signals from the force/torque sensors are incorporated into the control loop. The effectiveness of the proposed approach is well demonstrated with the experiments on several representative bimanual transportation tasks.
- 3 April 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cognitive and Developmental Systems
- Vol. 16 (1) , 251-265
A Distributed Dynamic Framework to Allocate Collaborative Tasks Based on Capability Matching in Heterogeneous Multirobot Systems
Collaboration among a group of robots with heterogeneous capabilities is an important research problem that enables to combine different robot functionalities, and thus, conducts complex tasks that may be difficult to achieve by a single robot with limited resources. In this paper, we propose a new distributed task allocation framework based on the capability matching of heterogeneous robots. The framework is composed of an ontological dynamic knowledge graph model and a hardware control scheme to model the capability and optimize resource utilization for collaborative tasks. We introduce an intuitive hardware control scheme based on a dynamic knowledge graph that resolves possible conflicts between the hardware control of different types of robots. Action sequences are produced by a task and motion planning algorithm to collaboratively perform the assigned task. The performance of the proposed methodology is evaluated by both simulations and hardware experiments.
- 3 April 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cognitive and Developmental Systems
- Vol. 16 (1) , 251-265
A Distributed Dynamic Framework to Allocate Collaborative Tasks Based on Capability Matching in Heterogeneous Multirobot Systems
Collaboration among a group of robots with heterogeneous capabilities is an important research problem that enables to combine different robot functionalities, and thus, conducts complex tasks that may be difficult to achieve by a single robot with limited resources. In this paper, we propose a new distributed task allocation framework based on the capability matching of heterogeneous robots. The framework is composed of an ontological dynamic knowledge graph model and a hardware control scheme to model the capability and optimize resource utilization for collaborative tasks. We introduce an intuitive hardware control scheme based on a dynamic knowledge graph that resolves possible conflicts between the hardware control of different types of robots. Action sequences are produced by a task and motion planning algorithm to collaboratively perform the assigned task. The performance of the proposed methodology is evaluated by both simulations and hardware experiments.
- 29 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Human-Machine Systems
- Vol. 53 (3) , 619-628
A Multisensor Interface to Improve the Learning Experience in Arc Welding Training Tasks
This article presents the development of a multisensor user interface to facilitate the instruction of arc welding tasks. Traditional methods to acquire hand-eye coordination skills are typically conducted through one-to-one instruction, where trainees must wear protective helmets and conduct several tests. These approaches are inefficient as the harmful light emitted from the electric arc impedes the close monitoring of the process. Practitioners can only observe a small bright spot. To tackle these problems, recent training approaches have leveraged virtual reality to safely simulate the process and visualize the geometry of the workpieces. However, the synthetic nature of these types of simulation platforms reduces their effectiveness as they fail to comprise actual welding interactions with the environment, which hinders the trainees' learning process. To provide users with a real welding experience, we have developed a new multisensor extended reality platform for arc welding training. Our system is composed of: 1) An HDR camera, monitoring the real welding spot in real time. 2) A depth sensor, capturing the 3-D geometry of the scene; and 3) A head-mounted VR display, visualizing the process safely. Our innovative platform provides users with a “bot trainer,” virtual cues of the seam geometry, automatic spot tracking, and performance scores. To validate the platform's feasibility, we conduct extensive experiments with several welding training tasks. We show that compared with the traditional training practice and recent virtual reality approaches, our automated multisensor method achieves better performances in terms of accuracy, learning curve, and effectiveness.
- 29 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Human-Machine Systems
- Vol. 53 (3) , 619-628
A Multisensor Interface to Improve the Learning Experience in Arc Welding Training Tasks
This article presents the development of a multisensor user interface to facilitate the instruction of arc welding tasks. Traditional methods to acquire hand-eye coordination skills are typically conducted through one-to-one instruction, where trainees must wear protective helmets and conduct several tests. These approaches are inefficient as the harmful light emitted from the electric arc impedes the close monitoring of the process. Practitioners can only observe a small bright spot. To tackle these problems, recent training approaches have leveraged virtual reality to safely simulate the process and visualize the geometry of the workpieces. However, the synthetic nature of these types of simulation platforms reduces their effectiveness as they fail to comprise actual welding interactions with the environment, which hinders the trainees' learning process. To provide users with a real welding experience, we have developed a new multisensor extended reality platform for arc welding training. Our system is composed of: 1) An HDR camera, monitoring the real welding spot in real time. 2) A depth sensor, capturing the 3-D geometry of the scene; and 3) A head-mounted VR display, visualizing the process safely. Our innovative platform provides users with a “bot trainer,” virtual cues of the seam geometry, automatic spot tracking, and performance scores. To validate the platform's feasibility, we conduct extensive experiments with several welding training tasks. We show that compared with the traditional training practice and recent virtual reality approaches, our automated multisensor method achieves better performances in terms of accuracy, learning curve, and effectiveness.
- 2 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8 (3) , 1643-1650
Self-Reconfigurable Soft-Rigid Mobile Agent With Variable Stiffness and Adaptive Morphology
In this letter, we propose a novel design of a hybrid mobile robot with controllable stiffness and deformable shape. Compared to conventional mobile agents, our system can switch between rigid and compliant phases by solidifying or melting Field's metal in its structure and, thus, alter the shape through the motion of its active components. In the soft state, the robot's main body can bend into circular arcs, which enables it to conform to surrounding curved objects. This variable geometry of the robot creates new motion modes which cannot be described by standard (i.e., fixed geometry) models. To this end, we develop a unified mathematical model that captures the differential kinematics of both rigid and soft states. An optimised control strategy is further proposed to select the most appropriate phase states and motion modes needed to reach a target pose-shape configuration. The performance of our new method is validated with numerical simulations and experiments conducted on a prototype system.
- 2 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8 (3) , 1643-1650
Self-Reconfigurable Soft-Rigid Mobile Agent With Variable Stiffness and Adaptive Morphology
In this letter, we propose a novel design of a hybrid mobile robot with controllable stiffness and deformable shape. Compared to conventional mobile agents, our system can switch between rigid and compliant phases by solidifying or melting Field's metal in its structure and, thus, alter the shape through the motion of its active components. In the soft state, the robot's main body can bend into circular arcs, which enables it to conform to surrounding curved objects. This variable geometry of the robot creates new motion modes which cannot be described by standard (i.e., fixed geometry) models. To this end, we develop a unified mathematical model that captures the differential kinematics of both rigid and soft states. An optimised control strategy is further proposed to select the most appropriate phase states and motion modes needed to reach a target pose-shape configuration. The performance of our new method is validated with numerical simulations and experiments conducted on a prototype system.