Tianbo Liu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Tianbo Liu has a total of 15 co-authors. Here are the top co-authors by number of shared publications:
- Shaojie Shen
Hong Kong University of Science and Technology, Hong Kong, Hong Kong
5 shared publications Shaojie Shen
Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong
4 shared publications- Jing Chen
Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China
2 shared publications - Chi Zhang
University of Electronic Science and Technology of China, Chengdu, China
1 shared publication - Dit-Yan Yeung
Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Sai Kung, Hong Kong
1 shared publication - Fei Gao
Zhejiang University
1 shared publication
Latest Publications
- Open Access
Learning Unmanned Aerial Vehicle Control for Autonomous Target Following
While deep reinforcement learning (RL) methods have achieved unprecedented successes in a range of challenging problems, their applicability has been mainly limited to simulation or game domains due to the high sample complexity of the trial-and-error learning process. However, real-world robotic applications often need a data-efficient learning process with safety-critical constraints. In this paper, we consider the challenging problem of learning unmanned aerial vehicle (UAV) control for tracking a moving target. To acquire a strategy that combines perception and control, we represent the policy by a convolutional neural network. We develop a hierarchical approach that combines a model-free policy gradient method with a conventional feedback proportional-integral-derivative (PID) controller to enable stable learning without catastrophic failure. The neural network is trained by a combination of supervised learning from raw images and reinforcement learning from games of self-play. We show that the proposed approach can learn a target following policy in a simulator efficiently and the learned behavior can be successfully transferred to the DJI quadrotor platform for real-world UAV control.
Open AccessLearning Unmanned Aerial Vehicle Control for Autonomous Target Following
While deep reinforcement learning (RL) methods have achieved unprecedented successes in a range of challenging problems, their applicability has been mainly limited to simulation or game domains due to the high sample complexity of the trial-and-error learning process. However, real-world robotic applications often need a data-efficient learning process with safety-critical constraints. In this paper, we consider the challenging problem of learning unmanned aerial vehicle (UAV) control for tracking a moving target. To acquire a strategy that combines perception and control, we represent the policy by a convolutional neural network. We develop a hierarchical approach that combines a model-free policy gradient method with a conventional feedback proportional-integral-derivative (PID) controller to enable stable learning without catastrophic failure. The neural network is trained by a combination of supervised learning from raw images and reinforcement learning from games of self-play. We show that the proposed approach can learn a target following policy in a simulator efficiently and the learned behavior can be successfully transferred to the DJI quadrotor platform for real-world UAV control.
- 11 July 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 2 (4) , 2224-2231
Spline-Based Initialization of Monocular Visual–Inertial State Estimators at High Altitude
Due to the lack of direct distance measurements, robust and accurate state estimation at high altitude but GPS-denied environments is a challenging task. A possible solution is monocular visual-inertial estimators, in which visual and inertial measurements are properly fused to recover the metric estimates. However, these estimators suffer from initialization under poor numerical conditioning or even degeneration, due to difficulties in retrieving observations of visual features with sufficient parallax, and the excessive period of inertial measurement integration. In this letter, we introduce the joint formulation into the spline-based high altitude estimator initialization method for monocular visual-inertial navigation system, in which the fitting of the spline and the alignment of visual measurements and inertial measurements are jointly optimized to recover metric estimates. The method ensures that sufficient excitation is contained in the inertial measurements when initialized, which eliminates the numerical issues. Compared with the work of Liu and Shen, the joint formulation makes our initialization method insensitive to the choice of spline parameters. Thus, the adaptivity to various environments and motions is obtained, as well as higher accuracy. The method is applicable for both loosely coupled and tightly coupled visual-inertial estimators. Extensive experiments are conducted to validate our approach.
- 11 July 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 2 (4) , 2224-2231
Spline-Based Initialization of Monocular Visual–Inertial State Estimators at High Altitude
Due to the lack of direct distance measurements, robust and accurate state estimation at high altitude but GPS-denied environments is a challenging task. A possible solution is monocular visual-inertial estimators, in which visual and inertial measurements are properly fused to recover the metric estimates. However, these estimators suffer from initialization under poor numerical conditioning or even degeneration, due to difficulties in retrieving observations of visual features with sufficient parallax, and the excessive period of inertial measurement integration. In this letter, we introduce the joint formulation into the spline-based high altitude estimator initialization method for monocular visual-inertial navigation system, in which the fitting of the spline and the alignment of visual measurements and inertial measurements are jointly optimized to recover metric estimates. The method ensures that sufficient excitation is contained in the inertial measurements when initialized, which eliminates the numerical issues. Compared with the work of Liu and Shen, the joint formulation makes our initialization method insensitive to the choice of spline parameters. Thus, the adaptivity to various environments and motions is obtained, as well as higher accuracy. The method is applicable for both loosely coupled and tightly coupled visual-inertial estimators. Extensive experiments are conducted to validate our approach.
- Top Cited Papers
- 4 July 2017
- Published by Wiley in Journal of Field Robotics
- Vol. 35 (1) , 23-51
Autonomous aerial navigation using monocular visual‐inertial fusion
Autonomous micro aerial vehicles (MAVs) have cost and mobility benefits, making them ideal robotic platforms for applications including aerial photography, surveillance, and search and rescue. As the platform scales down, MAVs become more capable of operating in confined environments, but it also introduces significant size and payload constraints. A monocular visual‐inertial navigation system (VINS), consisting only of an inertial measurement unit (IMU) and a camera, becomes the most suitable sensor suite in this case, thanks to its light weight and small footprint. In fact, it is the minimum sensor suite allowing autonomous flight with sufficient environmental awareness. In this paper, we show that it is possible to achieve reliable online autonomous navigation using monocular VINS. Our system is built on a customized quadrotor testbed equipped with a fisheye camera, a low‐cost IMU, and heterogeneous onboard computing resources. The backbone of our system is a highly accurate optimization‐based monocular visual‐inertial state estimator with online initialization and self‐extrinsic calibration. An onboard GPU‐based monocular dense mapping module that conditions on the estimated pose provides wide‐angle situational awareness. Finally, an online trajectory planner that operates directly on the incrementally built three‐dimensional map guarantees safe navigation through cluttered environments. Extensive experimental results are provided to validate individual system modules as well as the overall performance in both indoor and outdoor environments.
- 4 July 2017
- Published by Wiley in Journal of Field Robotics
- Vol. 35 (1) , 23-51
Top Cited PapersAutonomous aerial navigation using monocular visual‐inertial fusion
Autonomous micro aerial vehicles (MAVs) have cost and mobility benefits, making them ideal robotic platforms for applications including aerial photography, surveillance, and search and rescue. As the platform scales down, MAVs become more capable of operating in confined environments, but it also introduces significant size and payload constraints. A monocular visual‐inertial navigation system (VINS), consisting only of an inertial measurement unit (IMU) and a camera, becomes the most suitable sensor suite in this case, thanks to its light weight and small footprint. In fact, it is the minimum sensor suite allowing autonomous flight with sufficient environmental awareness. In this paper, we show that it is possible to achieve reliable online autonomous navigation using monocular VINS. Our system is built on a customized quadrotor testbed equipped with a fisheye camera, a low‐cost IMU, and heterogeneous onboard computing resources. The backbone of our system is a highly accurate optimization‐based monocular visual‐inertial state estimator with online initialization and self‐extrinsic calibration. An onboard GPU‐based monocular dense mapping module that conditions on the estimated pose provides wide‐angle situational awareness. Finally, an online trajectory planner that operates directly on the incrementally built three‐dimensional map guarantees safe navigation through cluttered environments. Extensive experimental results are provided to validate individual system modules as well as the overall performance in both indoor and outdoor environments.
- 1 May 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 55, 4544-4551
High altitude monocular visual-inertial state estimation: Initialization and sensor fusion
Obtaining reliable state estimates at high altitude but GPS-denied environments, such as between high-rise buildings or in the middle of deep canyons, is known to be challenging, due to the lack of direct distance measurements. Monocular visual-inertial systems provide a possible way to recover the metric distance through proper integration of visual and inertial measurements. However, the nonlinear optimization problem for state estimation suffers from poor numerical conditioning or even degeneration, due to difficulties in obtaining observations of visual features with sufficient parallax, and the excessive period of inertial measurement integration. In this paper, we propose a spline-based high altitude estimator initialization method for monocular visual-inertial navigation system (VINS) with special attention to the numerical issues. Our formulation takes only inertial measurements that contain sufficient excitation, and drops uninformative measurements such as those obtained during hovering. In addition, our method explicitly reduces the number of parameters to be estimated in order to achieve earlier convergence. Based on the initialization results, a complete closed-loop system is constructed for high altitude navigation. Extensive experiments are conducted to validate our approach.
- 1 May 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 55, 4544-4551
High altitude monocular visual-inertial state estimation: Initialization and sensor fusion
Obtaining reliable state estimates at high altitude but GPS-denied environments, such as between high-rise buildings or in the middle of deep canyons, is known to be challenging, due to the lack of direct distance measurements. Monocular visual-inertial systems provide a possible way to recover the metric distance through proper integration of visual and inertial measurements. However, the nonlinear optimization problem for state estimation suffers from poor numerical conditioning or even degeneration, due to difficulties in obtaining observations of visual features with sufficient parallax, and the excessive period of inertial measurement integration. In this paper, we propose a spline-based high altitude estimator initialization method for monocular visual-inertial navigation system (VINS) with special attention to the numerical issues. Our formulation takes only inertial measurements that contain sufficient excitation, and drops uninformative measurements such as those obtained during hovering. In addition, our method explicitly reduces the number of parameters to be estimated in order to achieve earlier convergence. Based on the initialization results, a complete closed-loop system is constructed for high altitude navigation. Extensive experiments are conducted to validate our approach.
- 26 January 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 2 (3) , 1256-1263
Model-Based Global Localization for Aerial Robots Using Edge Alignment
Robust state estimation is the core capability for autonomous aerial robots operating in complex environments. Global navigation satellite system and visual odometry/SLAM are popular methods for state estimation. However, there exist scenarios, such as when flying between tall buildings or in the middle of deep canyons, that all these methods fail due to obstructed sky view and high altitude. in this letter, inspired by the fact that offline-generated three-dimensional (3-D) models of cities and natural scenes are readily available, we propose a global localization method for aerial robots by using 3-D models and measurements from a monocular fisheye camera and an inertial measurement unit (IMU). Due to the fact that 3-D models are generated by different cameras at different times, traditional feature-based or direct registration methods usually fail to perform, we therefore propose to use an edge alignment-based method for image-to-model registration under strong changes in lighting conditions and camera characteristics. We additionally aid our model-based localization with electronic image stabilization for better tracking performance, and extended Kalman filter (EKF)-based sensor fusion for closed-loop control. Our method runs onboard an embedded computer in real time. We verify both local accuracy and global consistency of the proposed approach in real-world experiments with comparisons against ground truth. We also show closed-loop control using the proposed method as state feedback.
- 26 January 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 2 (3) , 1256-1263
Model-Based Global Localization for Aerial Robots Using Edge Alignment
Robust state estimation is the core capability for autonomous aerial robots operating in complex environments. Global navigation satellite system and visual odometry/SLAM are popular methods for state estimation. However, there exist scenarios, such as when flying between tall buildings or in the middle of deep canyons, that all these methods fail due to obstructed sky view and high altitude. in this letter, inspired by the fact that offline-generated three-dimensional (3-D) models of cities and natural scenes are readily available, we propose a global localization method for aerial robots by using 3-D models and measurements from a monocular fisheye camera and an inertial measurement unit (IMU). Due to the fact that 3-D models are generated by different cameras at different times, traditional feature-based or direct registration methods usually fail to perform, we therefore propose to use an edge alignment-based method for image-to-model registration under strong changes in lighting conditions and camera characteristics. We additionally aid our model-based localization with electronic image stabilization for better tracking performance, and extended Kalman filter (EKF)-based sensor fusion for closed-loop control. Our method runs onboard an embedded computer in real time. We verify both local accuracy and global consistency of the proposed approach in real-world experiments with comparisons against ground truth. We also show closed-loop control using the proposed method as state feedback.
- 1 December 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4984-4991
Self-calibrating multi-camera visual-inertial fusion for autonomous MAVs
We address the important problem of achieving robust and easy-to-deploy visual state estimation for micro aerial vehicles (MAVs) operating in complex environments. We use a sensor suite consisting of multiple cameras and an IMU to maximize perceptual awareness of the surroundings and provide sufficient redundancy against sensor failures. Our approach starts with an online initialization procedure that simultaneously estimates the transformation between each camera and the IMU, as well as the initial velocity and attitude of the platform, without any prior knowledge about the mechanical configuration of the sensor suite. Based on the initial calibrations, a tightly-coupled, optimization-based, generalized multi-camera-inertial fusion method runs onboard the MAV with online camera-IMU calibration refinement and identification of sensor failures. Our approach dynamically configures the system into monocular, stereo, or other multi-camera visual-inertial settings, with their respective perceptual advantages, based on the availability of visual measurements. We show that even under random camera failures, our method can be used for feedback control of the MAVs. We highlight our approach in challenging indoor-outdoor navigation tasks with large variations in vehicle height and speed, scene depth, and illumination.
- 1 December 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4984-4991
Self-calibrating multi-camera visual-inertial fusion for autonomous MAVs
We address the important problem of achieving robust and easy-to-deploy visual state estimation for micro aerial vehicles (MAVs) operating in complex environments. We use a sensor suite consisting of multiple cameras and an IMU to maximize perceptual awareness of the surroundings and provide sufficient redundancy against sensor failures. Our approach starts with an online initialization procedure that simultaneously estimates the transformation between each camera and the IMU, as well as the initial velocity and attitude of the platform, without any prior knowledge about the mechanical configuration of the sensor suite. Based on the initial calibrations, a tightly-coupled, optimization-based, generalized multi-camera-inertial fusion method runs onboard the MAV with online camera-IMU calibration refinement and identification of sensor failures. Our approach dynamically configures the system into monocular, stereo, or other multi-camera visual-inertial settings, with their respective perceptual advantages, based on the availability of visual measurements. We show that even under random camera failures, our method can be used for feedback control of the MAVs. We highlight our approach in challenging indoor-outdoor navigation tasks with large variations in vehicle height and speed, scene depth, and illumination.
- 1 October 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 446-453
Tracking a moving target in cluttered environments using a quadrotor
We address the challenging problem of tracking a moving target in cluttered environments using a quadrotor. Our online trajectory planning method generates smooth, dynamically feasible, and collision-free polynomial trajectories that follow a visually-tracked moving target. As visual observations of the target are obtained, the target trajectory can be estimated and used to predict the target motion for a short time horizon. We propose a formulation to embed both limited horizon tracking error and quadrotor control costs in the cost function for a quadratic programming (QP), while encoding both collision avoidance and dynamical feasibility as linear inequality constraints for the QP. Our method generates tracking trajectories in the order of milliseconds and is therefore suitable for online target tracking with a limited sensing range. We implement our approach on-board a quadrotor testbed equipped with cameras, a laser range finder, an IMU, and onboard computing. Statistical analysis, simulation, and real-world experiments are conducted to demonstrate the effectiveness of our approach.
- 1 October 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 446-453
Tracking a moving target in cluttered environments using a quadrotor
We address the challenging problem of tracking a moving target in cluttered environments using a quadrotor. Our online trajectory planning method generates smooth, dynamically feasible, and collision-free polynomial trajectories that follow a visually-tracked moving target. As visual observations of the target are obtained, the target trajectory can be estimated and used to predict the target motion for a short time horizon. We propose a formulation to embed both limited horizon tracking error and quadrotor control costs in the cost function for a quadratic programming (QP), while encoding both collision avoidance and dynamical feasibility as linear inequality constraints for the QP. Our method generates tracking trajectories in the order of milliseconds and is therefore suitable for online target tracking with a limited sensing range. We implement our approach on-board a quadrotor testbed equipped with cameras, a laser range finder, an IMU, and onboard computing. Statistical analysis, simulation, and real-world experiments are conducted to demonstrate the effectiveness of our approach.
- 1 May 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1499-1506
Aggressive quadrotor flight using dense visual-inertial fusion
In this work, we address the problem of aggressive flight of a quadrotor aerial vehicle using cameras and IMUs as the only sensing modalities. We present a fully integrated quadrotor system and demonstrate through online experiment the capability of autonomous flight with linear velocities up to 4.2 m/s, linear accelerations up to 9.6 m/s2, and angular velocities up to 245.1 degree/s. Central to our approach is a dense visual-inertial state estimator for reliable tracking of aggressive motions. An uncertainty-aware direct dense visual tracking module provides camera pose tracking that takes inverse depth uncertainty into account and is resistant to motion blur. Measurements from IMU pre-integration and multi-constrained dense visual tracking are fused probabilistically using an optimization-based sensor fusion framework. Extensive statistical analysis and comparison are presented to verify the performance of the proposed approach. We also release our code as open-source ROS packages.
- 1 May 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1499-1506
Aggressive quadrotor flight using dense visual-inertial fusion
In this work, we address the problem of aggressive flight of a quadrotor aerial vehicle using cameras and IMUs as the only sensing modalities. We present a fully integrated quadrotor system and demonstrate through online experiment the capability of autonomous flight with linear velocities up to 4.2 m/s, linear accelerations up to 9.6 m/s2, and angular velocities up to 245.1 degree/s. Central to our approach is a dense visual-inertial state estimator for reliable tracking of aggressive motions. An uncertainty-aware direct dense visual tracking module provides camera pose tracking that takes inverse depth uncertainty into account and is resistant to motion blur. Measurements from IMU pre-integration and multi-constrained dense visual tracking are fused probabilistically using an optimization-based sensor fusion framework. Extensive statistical analysis and comparison are presented to verify the performance of the proposed approach. We also release our code as open-source ROS packages.
- 1 May 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1476-1483
Online generation of collision-free trajectories for quadrotor flight in unknown cluttered environments
We present an online method for generating collision-free trajectories for autonomous quadrotor flight through cluttered environments. We consider the real-world scenario that the quadrotor aerial robot is equipped with limited sensing and operates in initially unknown environments. During flight, an octree-based environment representation is incrementally built using onboard sensors. Utilizing efficient operations in the octree data structure, we are able to generate free-space flight corridors consisting of large overlapping 3-D grids in an online fashion. A novel optimization-based method then generates smooth trajectories that both are bounded entirely within the safe flight corridor and satisfy higher order dynamical constraints. Our method computes valid trajectories within fractions of a second on a moderately fast computer, thus permitting online re-generation of trajectories for reaction to new obstacles. We build a complete quadrotor testbed with onboard sensing, state estimation, mapping, and control, and integrate the proposed method to show online navigation through complex unknown environments.
- 1 May 2016
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1476-1483
Online generation of collision-free trajectories for quadrotor flight in unknown cluttered environments
We present an online method for generating collision-free trajectories for autonomous quadrotor flight through cluttered environments. We consider the real-world scenario that the quadrotor aerial robot is equipped with limited sensing and operates in initially unknown environments. During flight, an octree-based environment representation is incrementally built using onboard sensors. Utilizing efficient operations in the octree data structure, we are able to generate free-space flight corridors consisting of large overlapping 3-D grids in an online fashion. A novel optimization-based method then generates smooth trajectories that both are bounded entirely within the safe flight corridor and satisfy higher order dynamical constraints. Our method computes valid trajectories within fractions of a second on a moderately fast computer, thus permitting online re-generation of trajectories for reaction to new obstacles. We build a complete quadrotor testbed with onboard sensing, state estimation, mapping, and control, and integrate the proposed method to show online navigation through complex unknown environments.