Jinyu Miao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Jinyu Miao has a total of 38 co-authors. Here are the top co-authors by number of shared publications:
- Kun Jiang
Tsinghua University
6 shared publications - Diange Yang
Tsinghua University
5 shared publications - Tuopu Wen
School of Vehicle and Mobility, Tsinghua University, Beijing, China
5 shared publications - Mengmeng Yang
Tsinghua University
4 shared publications - Benny Wijaya
School of Vehicle and Mobility, Tsinghua University, Beijing, China
3 shared publications - Jin Huang
Tsinghua University
3 shared publications
Latest Publications
- 1 December 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (12) , 14013-14020
nuTruck: Benchmarking Autonomous Driving Planning for Distributed Electric-Drive Trucks
- 1 December 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (12) , 14013-14020
nuTruck: Benchmarking Autonomous Driving Planning for Distributed Electric-Drive Trucks
- 2 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. PP, 1-8
Residual Learning Towards High-Fidelity Vehicle Dynamics Modeling With Transformer
The vehicle dynamics model serves as a vital component of autonomous driving systems, as it describes the temporal changes in vehicle state. Traditional physics-based methods employ mathematical formulae to model vehicle dynamics, but they are unable to adequately describe complex vehicle systems due to the simplifications they entail. The deep learning-based methods directly regress vehicle dynamics, but their performance and generalization capabilities still require further enhancement. In this letter, we address these problems by proposing a vehicle dynamics correction system that leverages deep neural networks to correct the state residuals of a physical model instead of directly estimating the states. This system greatly reduces the difficulty of network learning and thus improves the estimation accuracy of vehicle dynamics. Furthermore, we have developed a novel Transformer-based dynamics residual correction network, DyTR. This network implicitly represents state residuals as high-dimensional queries, and iteratively updates the estimated residuals by interacting with dynamics state features. Simulation experiments and real-world scaled-vehicle testing demonstrate the proposed vehicle dynamics correction system works much better than the physics-based vehicle model, and our proposed DyTR model achieves the state-of-the-art performance.
- 2 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. PP, 1-8
Residual Learning Towards High-Fidelity Vehicle Dynamics Modeling With Transformer
The vehicle dynamics model serves as a vital component of autonomous driving systems, as it describes the temporal changes in vehicle state. Traditional physics-based methods employ mathematical formulae to model vehicle dynamics, but they are unable to adequately describe complex vehicle systems due to the simplifications they entail. The deep learning-based methods directly regress vehicle dynamics, but their performance and generalization capabilities still require further enhancement. In this letter, we address these problems by proposing a vehicle dynamics correction system that leverages deep neural networks to correct the state residuals of a physical model instead of directly estimating the states. This system greatly reduces the difficulty of network learning and thus improves the estimation accuracy of vehicle dynamics. Furthermore, we have developed a novel Transformer-based dynamics residual correction network, DyTR. This network implicitly represents state residuals as high-dimensional queries, and iteratively updates the estimated residuals by interacting with dynamics state features. Simulation experiments and real-world scaled-vehicle testing demonstrate the proposed vehicle dynamics correction system works much better than the physics-based vehicle model, and our proposed DyTR model achieves the state-of-the-art performance.
- 19 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 10 (4) , 2519-2550
A Survey on Monocular Re-Localization: From the Perspective of Scene Map Representation
Monocular Re-Localization (MRL) is an essential component in autonomous applications, estimating 6 degree-of-freedom ego poses w.r.t. the scene map using monocular images. Over the past few decades, there has been notable advancement in the field of MRL techniques. Several works have achieved remarkable success in terms of both the accuracy and reliability of localization. In MRL, scene maps are represented in diverse forms, and they dictate the functionality and efficacy of MRL methods. However, to the best of our knowledge, existing surveys do not provide systematic reviews about the correlation between MRL solutions and their used scene map representation. This survey fills the gap by comprehensively reviewing MRL methods from such a perspective, promoting further research. 1) We commence by delving into the problem definition of MRL, exploring current challenges, and comparing ours with existing surveys. 2) MRL methods are categorized and reviewed into five classes based on the representation forms of the utilized map, i.e., geo-tagged frames, visual landmarks, point clouds, vectorized semantic maps, and neural network-based maps. 3) To quantitatively and fairly compare MRL methods using different maps, we introduce some public datasets and evaluate the performances of several advanced MRL approaches. The strengths and shortcomings of MRL methods using various maps are analyzed. 4) Finally, we introduce some topics of interest in this field and provide our individual viewpoints. This survey can serve as a valuable resource for MRL, and a summary of this survey is accessible to the community at: github.com/jinyummiao/map-in-mono-reloc
- 19 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 10 (4) , 2519-2550
A Survey on Monocular Re-Localization: From the Perspective of Scene Map Representation
Monocular Re-Localization (MRL) is an essential component in autonomous applications, estimating 6 degree-of-freedom ego poses w.r.t. the scene map using monocular images. Over the past few decades, there has been notable advancement in the field of MRL techniques. Several works have achieved remarkable success in terms of both the accuracy and reliability of localization. In MRL, scene maps are represented in diverse forms, and they dictate the functionality and efficacy of MRL methods. However, to the best of our knowledge, existing surveys do not provide systematic reviews about the correlation between MRL solutions and their used scene map representation. This survey fills the gap by comprehensively reviewing MRL methods from such a perspective, promoting further research. 1) We commence by delving into the problem definition of MRL, exploring current challenges, and comparing ours with existing surveys. 2) MRL methods are categorized and reviewed into five classes based on the representation forms of the utilized map, i.e., geo-tagged frames, visual landmarks, point clouds, vectorized semantic maps, and neural network-based maps. 3) To quantitatively and fairly compare MRL methods using different maps, we introduce some public datasets and evaluate the performances of several advanced MRL approaches. The strengths and shortcomings of MRL methods using various maps are analyzed. 4) Finally, we introduce some topics of interest in this field and provide our individual viewpoints. This survey can serve as a valuable resource for MRL, and a summary of this survey is accessible to the community at: github.com/jinyummiao/map-in-mono-reloc
- 29 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (1) , 803-810
Poses as Queries: End-to-End Image-to-LiDAR Map Localization With Transformers
High-precision vehicle localization with commercial setups is a crucial technique for high-level autonomous driving tasks. As a newly emerged approach, monocular localization in LiDAR map achieves promising balance between cost and accuracy, but estimating pose by finding correspondences between such cross-modal sensor data is challenging, thereby damaging the localization accuracy. In this letter, we address the problem by proposing a novel Transformer-based neural network to register 2D images into 3D LiDAR map in an end-to-end manner. We first implicitly represent poses as high-dimensional feature vectors called pose queries and gradually optimize poses by interacting with the retrieved relevant information from cross-modal features using attention mechanism in a proposed POse Estimator Transformer (POET) module. Moreover, we apply a multiple hypotheses aggregation method that estimates the final poses by performing parallel optimization on multiple randomly initialized pose queries to reduce the network uncertainty. Comprehensive analysis and experimental results on public benchmarks conclude that the proposed image-to-LiDAR map localization network could achieve state-of-the-art performances in challenging cross-modal localization tasks.
- 29 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (1) , 803-810
Poses as Queries: End-to-End Image-to-LiDAR Map Localization With Transformers
High-precision vehicle localization with commercial setups is a crucial technique for high-level autonomous driving tasks. As a newly emerged approach, monocular localization in LiDAR map achieves promising balance between cost and accuracy, but estimating pose by finding correspondences between such cross-modal sensor data is challenging, thereby damaging the localization accuracy. In this letter, we address the problem by proposing a novel Transformer-based neural network to register 2D images into 3D LiDAR map in an end-to-end manner. We first implicitly represent poses as high-dimensional feature vectors called pose queries and gradually optimize poses by interacting with the retrieved relevant information from cross-modal features using attention mechanism in a proposed POse Estimator Transformer (POET) module. Moreover, we apply a multiple hypotheses aggregation method that estimates the final poses by performing parallel optimization on multiple randomly initialized pose queries to reduce the network uncertainty. Comprehensive analysis and experimental results on public benchmarks conclude that the proposed image-to-LiDAR map localization network could achieve state-of-the-art performances in challenging cross-modal localization tasks.
- 23 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8 (12) , 8239-8246
SGFNet: Segmentation Guided Fusion Network for 3D Object Detection
The self-driving application requires accurate 3D object detection as it is essential in several tasks, such as path and motion planning. However, up until this point, fusion-based detectors with cameras and LiDAR sensors have always been inferior to LiDAR-only detectors. This can be attributed to the dual scene representation problem caused by the differentiated modality of LiDAR points and images. Moreover, the projection of the image pixels is not guaranteed to reach its point cloud counterparts due to the sparsity of the points, losing image content in the fusion process. Bearing these in mind, we propose Segmentation Guided Fusion Network (SGFNet), an efficient multi-sensor fusion-based 3D object detector. It first separates feature extractions of images and points with unified high-dimensional feature representation through the novel-proposed auxiliary foreground segmentation head, and then projects hierarchical feature maps instead of the raw image pixels onto points to obtain the unified feature map, achieving consistent data modality. Such image feature maps are with several spatial resolutions to keep more image content during the projection process. Finally, the unified feature map is fed into a fusion-based region proposal module and bounding box regression head to generate accurate 3D bounding boxes. Extensive experiments conducted on KITTI and nuScenes datasets demonstrate that SGFNet achieves competitive performance on fusion-based 3D object detection tasks and reports a novel state-of-the-art in terms of 3D average precision metric.
- 23 October 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 8 (12) , 8239-8246
SGFNet: Segmentation Guided Fusion Network for 3D Object Detection
The self-driving application requires accurate 3D object detection as it is essential in several tasks, such as path and motion planning. However, up until this point, fusion-based detectors with cameras and LiDAR sensors have always been inferior to LiDAR-only detectors. This can be attributed to the dual scene representation problem caused by the differentiated modality of LiDAR points and images. Moreover, the projection of the image pixels is not guaranteed to reach its point cloud counterparts due to the sparsity of the points, losing image content in the fusion process. Bearing these in mind, we propose Segmentation Guided Fusion Network (SGFNet), an efficient multi-sensor fusion-based 3D object detector. It first separates feature extractions of images and points with unified high-dimensional feature representation through the novel-proposed auxiliary foreground segmentation head, and then projects hierarchical feature maps instead of the raw image pixels onto points to obtain the unified feature map, achieving consistent data modality. Such image feature maps are with several spatial resolutions to keep more image content during the projection process. Finally, the unified feature map is fed into a fusion-based region proposal module and bounding box regression head to generate accurate 3D bounding boxes. Extensive experiments conducted on KITTI and nuScenes datasets demonstrate that SGFNet achieves competitive performance on fusion-based 3D object detection tasks and reports a novel state-of-the-art in terms of 3D average precision metric.
- 23 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 7 (3) , 7722-7729
Roadside HD Map Object Reconstruction Using Monocular Camera
The traditional HD map production based on mapping vehicles equipped with expensive sensors (e.g. LiDAR) struggles to keep the frequently changing HD maps up-to-date, thus calling for great research efforts in the HD map reconstruction using low-cost vision sensors. However, most existing works in this area limit their focus on reconstructing the landmarks on the road surface. This letter widens this limited scope by focusing on the roadside object reconstruction with a monocular camera. We propose a novel object-level reconstruction framework that boasts two advantageous features: 1) a novel 3D model estimation method to directly reconstruct the specific roadside objects in a vectorized format using their prior geometric knowledge, and 2) a novel data association method to solve the complex tracking problem of roadside objects that often yield thin and coherent observations. The proposed framework is evaluated on both typical highway and urban scenarios in public KAIST Urban dataset. The results demonstrate that our algorithm can reconstruct the roadside objects with high accuracy and recall, and outperforms classic Structure from Motion (SfM) and deep-learning-based methods by a large margin.
- 23 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 7 (3) , 7722-7729
Roadside HD Map Object Reconstruction Using Monocular Camera
The traditional HD map production based on mapping vehicles equipped with expensive sensors (e.g. LiDAR) struggles to keep the frequently changing HD maps up-to-date, thus calling for great research efforts in the HD map reconstruction using low-cost vision sensors. However, most existing works in this area limit their focus on reconstructing the landmarks on the road surface. This letter widens this limited scope by focusing on the roadside object reconstruction with a monocular camera. We propose a novel object-level reconstruction framework that boasts two advantageous features: 1) a novel 3D model estimation method to directly reconstruct the specific roadside objects in a vectorized format using their prior geometric knowledge, and 2) a novel data association method to solve the complex tracking problem of roadside objects that often yield thin and coherent observations. The proposed framework is evaluated on both typical highway and urban scenarios in public KAIST Urban dataset. The results demonstrate that our algorithm can reconstruct the roadside objects with high accuracy and recall, and outperforms classic Structure from Motion (SfM) and deep-learning-based methods by a large margin.
- 10 June 2022
- Published by Wiley in Journal of Field Robotics
- Vol. 39 (7) , 1069-1084
Automatic vocabulary and graph verification for accurate loop closure detection
Localizing previously visited places during long‐term localization and mapping, that is, loop closure detection (LCD), is a crucial technique to correct accumulated inconsistencies. In common bag‐of‐words (BoW) model, a visual vocabulary is built to associate features for detecting loops. Currently, methods that build vocabularies off‐line determine scales of the vocabulary by trial‐and‐error, which results in unreasonable feature association. Moreover, the detection precision of the algorithm declines due to perceptual aliasing given that the BoW‐based method ignores the positions of visual features. To build the optimal vocabulary automatically and eliminate human heuristics, we propose a natural convergence criterion based on the comparison between the radii of nodes and the drifts of feature descriptors in vocabulary construction. Furthermore, a novel topological graph verification method is proposed for validating loop candidates, which can effectively distinguish visual ambiguities by involving geometrical position of features and thus improve the precision of LCD. Experiments on various public datasets verify the effectiveness of our proposed approach.
- 10 June 2022
- Published by Wiley in Journal of Field Robotics
- Vol. 39 (7) , 1069-1084
Automatic vocabulary and graph verification for accurate loop closure detection
Localizing previously visited places during long‐term localization and mapping, that is, loop closure detection (LCD), is a crucial technique to correct accumulated inconsistencies. In common bag‐of‐words (BoW) model, a visual vocabulary is built to associate features for detecting loops. Currently, methods that build vocabularies off‐line determine scales of the vocabulary by trial‐and‐error, which results in unreasonable feature association. Moreover, the detection precision of the algorithm declines due to perceptual aliasing given that the BoW‐based method ignores the positions of visual features. To build the optimal vocabulary automatically and eliminate human heuristics, we propose a natural convergence criterion based on the comparison between the radii of nodes and the drifts of feature descriptors in vocabulary construction. Furthermore, a novel topological graph verification method is proposed for validating loop candidates, which can effectively distinguish visual ambiguities by involving geometrical position of features and thus improve the precision of LCD. Experiments on various public datasets verify the effectiveness of our proposed approach.
- Top Cited Papers
- 3 March 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 3101-3112
ALIKE: Accurate and Lightweight Keypoint Detection and Descriptor Extraction
Existing methods detect the keypoints in a non-differentiable way, therefore they can not directly optimize the position of keypoints through back-propagation. To address this issue, we present a partially differentiable keypoint detection module, which outputs accurate sub-pixel keypoints. The reprojection loss is then proposed to directly optimize these sub-pixel keypoints, and the dispersity peak loss is presented for accurate keypoints regularization. We also extract the descriptors in a sub-pixel way, and they are trained with the stable neural reprojection error loss. Moreover, a lightweight network is designed for keypoint detection and descriptor extraction, which can run at 95 frames per second for 640x480 images on a commercial GPU. On homography estimation, camera pose estimation, and visual (re-)localization tasks, the proposed method achieves equivalent performance with the state-of-the-art approaches, while greatly reduces the inference time.
- 3 March 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 3101-3112
Top Cited PapersALIKE: Accurate and Lightweight Keypoint Detection and Descriptor Extraction
Existing methods detect the keypoints in a non-differentiable way, therefore they can not directly optimize the position of keypoints through back-propagation. To address this issue, we present a partially differentiable keypoint detection module, which outputs accurate sub-pixel keypoints. The reprojection loss is then proposed to directly optimize these sub-pixel keypoints, and the dispersity peak loss is presented for accurate keypoints regularization. We also extract the descriptors in a sub-pixel way, and they are trained with the stable neural reprojection error loss. Moreover, a lightweight network is designed for keypoint detection and descriptor extraction, which can run at 95 frames per second for 640x480 images on a commercial GPU. On homography estimation, camera pose estimation, and visual (re-)localization tasks, the proposed method achieves equivalent performance with the state-of-the-art approaches, while greatly reduces the inference time.
- 29 December 2020
- Published by Wiley in Computer Applications in Engineering Education
- Vol. 29 (5) , 1176-1190
Simulation for senior undergraduate education of robot engineering based on Webots
In this paper, a comprehensive simulation framework is presented for senior undergraduates of robot engineering based on open‐source simulation software, Webots. In our curriculum, a virtual quadruped robot is used to test different theories. First, students should design the physical structure of the robot and select proper drivers according to the requirements on its motion ability. Then, the kinematics and workspace of each leg should be analyzed. To overcome different terrains, terrain recognition, and motion planning methods are presented. Finally, the processes of building a virtual robot and its test environment are introduced. Simulation results demonstrate that it is possible to test different robot techniques in our proposed simulation framework. Teaching practice and feedback from students show their promotion of understanding of robot theories. Their interest in robots are also increased.
- 29 December 2020
- Published by Wiley in Computer Applications in Engineering Education
- Vol. 29 (5) , 1176-1190
Simulation for senior undergraduate education of robot engineering based on Webots
In this paper, a comprehensive simulation framework is presented for senior undergraduates of robot engineering based on open‐source simulation software, Webots. In our curriculum, a virtual quadruped robot is used to test different theories. First, students should design the physical structure of the robot and select proper drivers according to the requirements on its motion ability. Then, the kinematics and workspace of each leg should be analyzed. To overcome different terrains, terrain recognition, and motion planning methods are presented. Finally, the processes of building a virtual robot and its test environment are introduced. Simulation results demonstrate that it is possible to test different robot techniques in our proposed simulation framework. Teaching practice and feedback from students show their promotion of understanding of robot theories. Their interest in robots are also increased.