Ruixuan Liu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Ruixuan Liu has a total of 24 co-authors. Here are the top co-authors by number of shared publications:
- Changliu Liu
Carnegie Mellon University, Pittsburgh, PA, USA
10 shared publications - Kangle Deng
Roblox, San Mateo, USA
2 shared publications - Shaojun Xu
Department of Precision Instrument, Tsinghua University, Beijing, China
2 shared publications - Xusheng Luo
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA
2 shared publications - Alan Chen
Westlake High School, Austin, TX, USA
1 shared publication - Ava Pun
Carnegie Mellon University
1 shared publication
Latest Publications
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics & Automation Magazine
- Vol. 33 (3) , 2-13
Prompt-to-Product: Generative Assembly via Bimanual Manipulation
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics & Automation Magazine
- Vol. 33 (3) , 2-13
Prompt-to-Product: Generative Assembly via Bimanual Manipulation
- 13 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (10) , 10482-10489
Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Specifications
To enable non-experts to specify long-horizon, multi-robot collaborative tasks, language models are increasingly used to translate natural language commands into formal specifications. However, because translation can occur in multiple ways, such translations may lack accuracy or lead to inefficient multi-robot planning. Our key insight is that concise hierarchical specifications can simplify planning while remaining straightforward to derive from human instructions. We propose Nl2Hltl2Plan, a framework that translates natural language commands into hierarchical Linear Temporal Logic (LTL) and solves the corresponding planning problem. The translation involves two steps leveraging Large Language Models (LLMs). First, an LLM transforms instructions into a Hierarchical Task Tree, capturing logical and temporal relations. Next, a fine-tuned LLM converts sub-tasks into standard LTL formulas, which are aggregated into hierarchical specifications, with the lowest level corresponding to ordered robot actions. These specifications are then used with off-the-shelf planners. Our Nl2Hltl2Plan demonstrates the potential of LLMs in hierarchical reasoning for multi-robot task planning. Evaluations in simulation and real-world experiments with human participants show that Nl2Hltl2Plan outperforms existing methods, handling more complex instructions while achieving higher success rates and lower costs in task allocation and planning.
- 13 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (10) , 10482-10489
Nl2Hltl2Plan: Scaling Up Natural Language Understanding for Multi-Robots Through Hierarchical Temporal Logic Task Specifications
To enable non-experts to specify long-horizon, multi-robot collaborative tasks, language models are increasingly used to translate natural language commands into formal specifications. However, because translation can occur in multiple ways, such translations may lack accuracy or lead to inefficient multi-robot planning. Our key insight is that concise hierarchical specifications can simplify planning while remaining straightforward to derive from human instructions. We propose Nl2Hltl2Plan, a framework that translates natural language commands into hierarchical Linear Temporal Logic (LTL) and solves the corresponding planning problem. The translation involves two steps leveraging Large Language Models (LLMs). First, an LLM transforms instructions into a Hierarchical Task Tree, capturing logical and temporal relations. Next, a fine-tuned LLM converts sub-tasks into standard LTL formulas, which are aggregated into hierarchical specifications, with the lowest level corresponding to ordered robot actions. These specifications are then used with off-the-shelf planners. Our Nl2Hltl2Plan demonstrates the potential of LLMs in hierarchical reasoning for multi-robot task planning. Evaluations in simulation and real-world experiments with human participants show that Nl2Hltl2Plan outperforms existing methods, handling more complex instructions while achieving higher success rates and lower costs in task allocation and planning.
- 26 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (5) , 4882-4889
Physics-Aware Combinatorial Assembly Sequence Planning Using Data-Free Action Masking
Combinatorial assembly uses standardized unit primitives to build objects that satisfy user specifications. This paper studies assembly sequence planning (ASP) for physical combinatorial assembly. Given the shape of the desired object, the goal is to find a sequence of actions for placing unit primitives to build the target object. In particular, we aim to ensure the planned assembly sequence is physically executable. However, ASP for combinatorial assembly is particularly challenging due to its combinatorial nature. To address the challenge, we employ deep reinforcement learning to learn a construction policy for placing unit primitives sequentially to build the desired object. Specifically, we design an online physics-aware action mask that filters out invalid actions, which effectively guides policy learning and ensures violation-free deployment. In the end, we apply the proposed method to Lego assembly with more than 250 3D structures. The experiment results demonstrate that the proposed method plans physically valid assembly sequences to build all structures, achieving a $100\%$ success rate, whereas the best comparable baseline fails more than 40 structures. Our implementation is available at https://github.com/intelligent-control-lab/PhysicsAwareCombinatorialASP.
- 26 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (5) , 4882-4889
Physics-Aware Combinatorial Assembly Sequence Planning Using Data-Free Action Masking
Combinatorial assembly uses standardized unit primitives to build objects that satisfy user specifications. This paper studies assembly sequence planning (ASP) for physical combinatorial assembly. Given the shape of the desired object, the goal is to find a sequence of actions for placing unit primitives to build the target object. In particular, we aim to ensure the planned assembly sequence is physically executable. However, ASP for combinatorial assembly is particularly challenging due to its combinatorial nature. To address the challenge, we employ deep reinforcement learning to learn a construction policy for placing unit primitives sequentially to build the desired object. Specifically, we design an online physics-aware action mask that filters out invalid actions, which effectively guides policy learning and ensures violation-free deployment. In the end, we apply the proposed method to Lego assembly with more than 250 3D structures. The experiment results demonstrate that the proposed method plans physically valid assembly sequences to build all structures, achieving a $100\%$ success rate, whereas the best comparable baseline fails more than 40 structures. Our implementation is available at https://github.com/intelligent-control-lab/PhysicsAwareCombinatorialASP.
- 13 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (11) , 9383-9390
StableLego: Stability Analysis of Block Stacking Assembly
Structural stability is a necessary condition for successful construction of an assembly. However, designing a stable assembly requires a non-trivial effort since a slight variation in the design could significantly affect the structural stability. To address the challenge, this paper studies the stability of assembly structures, in particular, block stacking assembly. The paper proposes a new optimization formulation, which optimizes over force balancing equations, for inferring the structural stability of 3D block stacking structures. The proposed stability analysis is verified on hand-crafted Lego examples. The experiment results demonstrate that the proposed method can correctly predict whether the structure is stable. In addition, it outperforms the existing methods since it can accurately locate the weakest parts in the design, and more importantly, solve any given assembly structures. To further validate the proposed method, we provide StableLego: a comprehensive dataset including 50 k+ 3D objects with their Lego layouts. We test the proposed stability analysis and include the stability inference for each corresponding object in StableLego.
- 13 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (11) , 9383-9390
StableLego: Stability Analysis of Block Stacking Assembly
Structural stability is a necessary condition for successful construction of an assembly. However, designing a stable assembly requires a non-trivial effort since a slight variation in the design could significantly affect the structural stability. To address the challenge, this paper studies the stability of assembly structures, in particular, block stacking assembly. The paper proposes a new optimization formulation, which optimizes over force balancing equations, for inferring the structural stability of 3D block stacking structures. The proposed stability analysis is verified on hand-crafted Lego examples. The experiment results demonstrate that the proposed method can correctly predict whether the structure is stable. In addition, it outperforms the existing methods since it can accurately locate the weakest parts in the design, and more importantly, solve any given assembly structures. To further validate the proposed method, we provide StableLego: a comprehensive dataset including 50 k+ 3D objects with their Lego layouts. We test the proposed stability analysis and include the stability inference for each corresponding object in StableLego.
- 11 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (8) , 7182-7189
Decomposition-Based Hierarchical Task Allocation and Planning for Multi-Robots Under Hierarchical Temporal Logic Specifications
Past research into robotic planning with temporal logic specifications, notably Linear Temporal Logic (LTL), was largely based on a single formula for individual or groups of robots. But with increasing task complexity, LTL formulas unavoidably grow lengthy, complicating interpretation and specification generation, and straining the computational capacities of the planners. A recent development has been the hierarchical representation of LTL (Luo et al., 2024) that contains multiple temporal logic specifications, providing a more interpretable framework. However, the proposed planning algorithm assumes the independence of robots within each specification, limiting their application to multi-robot coordination with complex temporal constraints. In this work, we formulated a decomposition-based hierarchical framework. At the high level, each specification is first decomposed into a set of atomic sub-tasks. We further infer the temporal relations among the sub-tasks of different specifications to construct a task network. Subsequently, a Mixed Integer Linear Program is used to assign sub-tasks to various robots. At the lower level, domain-specific controllers are employed to execute sub-tasks. Our approach was experimentally applied to domains of navigation and manipulation. The simulation demonstrated that our approach can find better solutions using less runtimes.
- 11 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (8) , 7182-7189
Decomposition-Based Hierarchical Task Allocation and Planning for Multi-Robots Under Hierarchical Temporal Logic Specifications
Past research into robotic planning with temporal logic specifications, notably Linear Temporal Logic (LTL), was largely based on a single formula for individual or groups of robots. But with increasing task complexity, LTL formulas unavoidably grow lengthy, complicating interpretation and specification generation, and straining the computational capacities of the planners. A recent development has been the hierarchical representation of LTL (Luo et al., 2024) that contains multiple temporal logic specifications, providing a more interpretable framework. However, the proposed planning algorithm assumes the independence of robots within each specification, limiting their application to multi-robot coordination with complex temporal constraints. In this work, we formulated a decomposition-based hierarchical framework. At the high level, each specification is first decomposed into a set of atomic sub-tasks. We further infer the temporal relations among the sub-tasks of different specifications to construct a task network. Subsequently, a Mixed Integer Linear Program is used to assign sub-tasks to various robots. At the lower level, domain-specific controllers are employed to execute sub-tasks. Our approach was experimentally applied to domains of navigation and manipulation. The simulation demonstrated that our approach can find better solutions using less runtimes.
- Open Access
- 13 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8614-8619
Zero-Shot Transferable and Persistently Feasible Safe Control for High Dimensional Systems by Consistent Abstraction
Safety is critical in robotic tasks. Energy function based methods have been introduced to address the problem. To ensure safety in the presence of control limits, we need to design an energy function that results in persistently feasible safe control at all system states. However, designing such an energy function for high-dimensional nonlinear systems remains challenging. Considering the fact that there are redundant dynamics in high dimensional systems with respect to the safety specifications, this paper proposes a novel approach called abstract safe control. We propose a system abstraction method that enables the design of energy functions on a low-dimensional model. Then we can synthesize the energy function with respect to the low-dimensional model to ensure persistent feasibility. The resulting safe controller can be directly transferred to other systems with the same abstraction, e.g., when a robot arm holds different tools. The proposed approach is demonstrated on a 7-DoF robot arm (14 states) both in simulation and real-world. Our method always finds feasible control and achieves zero safety violations in 500 trials on 5 different systems.
- 13 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8614-8619
Open AccessZero-Shot Transferable and Persistently Feasible Safe Control for High Dimensional Systems by Consistent Abstraction
Safety is critical in robotic tasks. Energy function based methods have been introduced to address the problem. To ensure safety in the presence of control limits, we need to design an energy function that results in persistently feasible safe control at all system states. However, designing such an energy function for high-dimensional nonlinear systems remains challenging. Considering the fact that there are redundant dynamics in high dimensional systems with respect to the safety specifications, this paper proposes a novel approach called abstract safe control. We propose a system abstraction method that enables the design of energy functions on a low-dimensional model. Then we can synthesize the energy function with respect to the low-dimensional model to ensure persistent feasibility. The resulting safe controller can be directly transferred to other systems with the same abstraction, e.g., when a robot arm holds different tools. The proposed approach is demonstrated on a 7-DoF robot arm (14 states) both in simulation and real-world. Our method always finds feasible control and achieves zero safety violations in 500 trials on 5 different systems.
- Open Access
Robotic LEGO Assembly and Disassembly from Human Demonstration
This paper studies automatic prototyping using LEGO. To satisfy individual needs and self-sustainability, this paper presents a framework that learns the assembly and disassembly sequences from human demonstrations. In addition, a digital twin is developed to verify the correctness of robot learning before deploying to the real world. Moreover, an end-effector tool (EOT) is designed, which allows large industrial robots to easily manipulate LEGO bricks. The proposed system is deployed to a FANUC LR-mate 200id/7L robot. Experiments demonstrate that the proposed system can effectively learn the assembly and disassembly tasks from human demonstrations. And the learned tasks are realized by the FANUC robot.
Open AccessRobotic LEGO Assembly and Disassembly from Human Demonstration
This paper studies automatic prototyping using LEGO. To satisfy individual needs and self-sustainability, this paper presents a framework that learns the assembly and disassembly sequences from human demonstrations. In addition, a digital twin is developed to verify the correctness of robot learning before deploying to the real world. Moreover, an end-effector tool (EOT) is designed, which allows large industrial robots to easily manipulate LEGO bricks. The proposed system is deployed to a FANUC LR-mate 200id/7L robot. Experiments demonstrate that the proposed system can effectively learn the assembly and disassembly tasks from human demonstrations. And the learned tasks are realized by the FANUC robot.
- Open Access
Data Efficient Human Intention Prediction: Leveraging Neural Network Verification and Expert Guidance
Predicting human intention is critical to facilitating safe and efficient human-robot collaboration (HRC). However, it is challenging to build data-driven models for human intention prediction. One major challenge is due to the diversity and noise in human motion data. It is expensive to collect a massive motion dataset that comprehensively covers all possible scenarios, which leads to the scarcity of human motion data in certain scenarios, and therefore, causes difficulties in constructing robust and reliable intention predictors. To address the challenge, this paper proposes an iterative adversarial data augmentation (IADA) framework to learn neural network models from an insufficient amount of training data. The method uses neural network verification to identify the most "confusing" input samples and leverages expert guidance to safely and iteratively augment the training data with these samples. The proposed framework is applied to collected human datasets. The experiments demonstrate that our method can achieve more robust and accurate prediction performance compared to existing training methods.
Open AccessData Efficient Human Intention Prediction: Leveraging Neural Network Verification and Expert Guidance
Predicting human intention is critical to facilitating safe and efficient human-robot collaboration (HRC). However, it is challenging to build data-driven models for human intention prediction. One major challenge is due to the diversity and noise in human motion data. It is expensive to collect a massive motion dataset that comprehensively covers all possible scenarios, which leads to the scarcity of human motion data in certain scenarios, and therefore, causes difficulties in constructing robust and reliable intention predictors. To address the challenge, this paper proposes an iterative adversarial data augmentation (IADA) framework to learn neural network models from an insufficient amount of training data. The method uses neural network verification to identify the most "confusing" input samples and leverages expert guidance to safely and iteratively augment the training data with these samples. The proposed framework is applied to collected human datasets. The experiments demonstrate that our method can achieve more robust and accurate prediction performance compared to existing training methods.
- Open Access
- 28 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in TechRxiv
Jerk-bounded Position Controller with Real-Time Task Modification for Interactive Industrial Robots
This paper presents a jerk-bounded position control driver (JPC) for industrial robots. JPC provides a unified interface for tracking complex trajectories and is able to enforce dynamic constraints using motion-level control, without accessing servo-level control. Most importantly, JPC enables real-time trajectory modification. Users can overwrite the ongoing task with a new one without violating dynamic constraints. The proposed JPC is implemented and tested on the FANUC LR Mate 200id/7L robot with both artificially generated data and an interactive robot handover task. Experiments show that the proposed JPC can track complex trajectories accurately within jerk limits and seamlessly switch to new trajectory references before the ongoing task ends.
- 28 January 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in TechRxiv
Open AccessJerk-bounded Position Controller with Real-Time Task Modification for Interactive Industrial Robots
This paper presents a jerk-bounded position control driver (JPC) for industrial robots. JPC provides a unified interface for tracking complex trajectories and is able to enforce dynamic constraints using motion-level control, without accessing servo-level control. Most importantly, JPC enables real-time trajectory modification. Users can overwrite the ongoing task with a new one without violating dynamic constraints. The proposed JPC is implemented and tested on the FANUC LR Mate 200id/7L robot with both artificially generated data and an interactive robot handover task. Experiments show that the proposed JPC can track complex trajectories accurately within jerk limits and seamlessly switch to new trajectory references before the ongoing task ends.
- 25 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3725-3730
Human Motion Prediction Using Adaptable Recurrent Neural Networks and Inverse Kinematics
Human motion prediction, especially arm prediction, is critical to facilitate safe and efficient human-robot collaboration (HRC). This paper proposes a novel human motion prediction framework that combines a recurrent neural network (RNN) and inverse kinematics (IK) to predict human arm motion. A modified Kalman filter (MKF) is applied to adapt the model online. The proposed framework is tested on collected human motion data with up to 2 s prediction horizon. The experiments demonstrate that the proposed method improves the prediction accuracy by approximately 14% comparing to the state-of-art on seen situations. It stably adapts to unseen situations by keeping the maximum prediction error under 4 cm, which is 70% lower than other methods. Moreover, it is robust when the arm is partially occluded. The wrist prediction remains the same, while the elbow prediction has 20% less variation.
- 25 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3725-3730
Human Motion Prediction Using Adaptable Recurrent Neural Networks and Inverse Kinematics
Human motion prediction, especially arm prediction, is critical to facilitate safe and efficient human-robot collaboration (HRC). This paper proposes a novel human motion prediction framework that combines a recurrent neural network (RNN) and inverse kinematics (IK) to predict human arm motion. A modified Kalman filter (MKF) is applied to adapt the model online. The proposed framework is tested on collected human motion data with up to 2 s prediction horizon. The experiments demonstrate that the proposed method improves the prediction accuracy by approximately 14% comparing to the state-of-art on seen situations. It stably adapts to unseen situations by keeping the maximum prediction error under 4 cm, which is 70% lower than other methods. Moreover, it is robust when the arm is partially occluded. The wrist prediction remains the same, while the elbow prediction has 20% less variation.