Zhe Xuanyuan
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Zhe Xuanyuan has a total of 41 co-authors. Here are the top co-authors by number of shared publications:
- Long Chen
State Key Laboratory of Multimodal Artificial Intelligence Systems and the State Key Laboratory of Management and Control for Complex Systems, Chinese Academy of Sciences, Beijing, China
7 shared publications - Yuchen Li
Beijing Normal-Hong Kong Baptist University,Faculty of Science and Technology,Zhuhai,China,519087
7 shared publications - Yunfeng Ai
University of Chinese Academy of Sciences, Beijing, China
5 shared publications - Lingxi Li
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA
4 shared publications - Luxi Li
Faculty of Science and Technology, BNU-HKBU United International College, Zhuhai, China
4 shared publications - Siyu Teng
Hong Kong Baptist University
4 shared publications
Latest Publications
- 1 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/CAA Journal of Automatica Sinica
- Vol. 13 (3) , 501-520
Datasets, Metrics, Benchmarks and Future Research in Autonomous Driving: A Review
Data-driven autonomous driving is a hot topic in academic and industry research due to its impressive performance, flexible mobility, and reduced human intervention. However, the development of this technology relies heavily on large datasets that contain accurately annotated data, obtained through artificial or semi-automated strategies. Consequently, datasets play a crucial role in autonomous driving, and their characteristics significantly impact the effectiveness of algorithms. Currently, there are several diverse datasets available, such as KITTI and CityScape, that cover various tasks. However, researchers often overlook the unique features, similarities, and specificities of these datasets. Furthermore, to the best of our knowledge, there is a lack of survey articles focusing on special metrics and benchmark performance on different datasets in autonomous driving. Therefore, the purpose of this article is to analyze autonomous driving datasets, guide researchers on collecting and utilizing relevant datasets, summarize evaluation strategies, analyze benchmark performance, and provide future research points to enrich the autonomous driving community. We believe that this work will assist researchers in evaluating their data using suitable metrics and offer a fresh perspective on autonomous driving.
- 1 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/CAA Journal of Automatica Sinica
- Vol. 13 (3) , 501-520
Datasets, Metrics, Benchmarks and Future Research in Autonomous Driving: A Review
Data-driven autonomous driving is a hot topic in academic and industry research due to its impressive performance, flexible mobility, and reduced human intervention. However, the development of this technology relies heavily on large datasets that contain accurately annotated data, obtained through artificial or semi-automated strategies. Consequently, datasets play a crucial role in autonomous driving, and their characteristics significantly impact the effectiveness of algorithms. Currently, there are several diverse datasets available, such as KITTI and CityScape, that cover various tasks. However, researchers often overlook the unique features, similarities, and specificities of these datasets. Furthermore, to the best of our knowledge, there is a lack of survey articles focusing on special metrics and benchmark performance on different datasets in autonomous driving. Therefore, the purpose of this article is to analyze autonomous driving datasets, guide researchers on collecting and utilizing relevant datasets, summarize evaluation strategies, analyze benchmark performance, and provide future research points to enrich the autonomous driving community. We believe that this work will assist researchers in evaluating their data using suitable metrics and offer a fresh perspective on autonomous driving.
- 31 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Field Robotics
- Vol. 2, 131-146
Online Triangular Constraint Calibration for LiDAR and Cameras in Open-Pit Mines
Extrinsic calibration between a LiDAR and camera is a crucial task for intelligent vehicles and mobile robots. However, most of the current extrinsic calibration methods are offline and target-based, making it hard to achieve real-time calibration depending on the characteristics of the environments. Although a few online calibration strategies can solve the above drawbacks, they still rely on a large amount of labeled data and struggle to achieve satisfactory performance in extrinsic calibration in unstructured scenarios where valid targets are scarce. In this work, we propose an effective online calibration strategy for mining scenarios, one of the typical unstructured scenarios. Based on the analysis of mining area data, we adopt an approach that integrates image and point cloud segmentation to accomplish the entire calibration process. An initialization method supporting a single rigid target is proposed, effectively addressing the initialization failure caused by the scarcity of targets in mining areas. Incorporating the segmentation results from the right camera to establish a left-right camera-LiDAR triangular constraint effectively reduces translation errors. Temporal data with confidence scores are then used to update and optimize the extrinsic parameters. Our online extrinsic calibration method has been validated on mining scenarios data and has achieved less than 0.2 m in data collected from an SUV and 0.3 m from trucks in translation errors within 10 frames. Besides, we provide a detailed discussion on the challenges of calibration in mining areas and the limitations of the proposed method.
- 31 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Field Robotics
- Vol. 2, 131-146
Online Triangular Constraint Calibration for LiDAR and Cameras in Open-Pit Mines
Extrinsic calibration between a LiDAR and camera is a crucial task for intelligent vehicles and mobile robots. However, most of the current extrinsic calibration methods are offline and target-based, making it hard to achieve real-time calibration depending on the characteristics of the environments. Although a few online calibration strategies can solve the above drawbacks, they still rely on a large amount of labeled data and struggle to achieve satisfactory performance in extrinsic calibration in unstructured scenarios where valid targets are scarce. In this work, we propose an effective online calibration strategy for mining scenarios, one of the typical unstructured scenarios. Based on the analysis of mining area data, we adopt an approach that integrates image and point cloud segmentation to accomplish the entire calibration process. An initialization method supporting a single rigid target is proposed, effectively addressing the initialization failure caused by the scarcity of targets in mining areas. Incorporating the segmentation results from the right camera to establish a left-right camera-LiDAR triangular constraint effectively reduces translation errors. Temporal data with confidence scores are then used to update and optimize the extrinsic parameters. Our online extrinsic calibration method has been validated on mining scenarios data and has achieved less than 0.2 m in data collected from an SUV and 0.3 m from trucks in translation errors within 10 frames. Besides, we provide a detailed discussion on the challenges of calibration in mining areas and the limitations of the proposed method.
- 24 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (5) , 4831-4834
Embodied Intelligence in Mining: Leveraging Multi-Modal Large Language Models for Autonomous Driving in Mines
With computer technology advancing in both software and hardware, the benefits of embodied intelligence are becoming increasingly evident. This robust interactive learning model enables artificial intelligence (AI) to be more flexibly deployed across diverse fields. In recent years, the development of multi-modal large language models (LLMs) has further accelerated the progress of AI, prompting extensive research on how to leverage these advancements to enhance the field of autonomous driving. This perspective believes that embodied intelligence can significantly enhance the application of LLMs, analyzing the new opportunities brought to the mining industry, and emphasizing the potential of their integration to revolutionize various aspects of the field. Meanwhile, This perspective also examines the challenges of deploying embodied agents in mining, while emphasizing their promising future and offering insights into potential research and development avenues.
- 24 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (5) , 4831-4834
Embodied Intelligence in Mining: Leveraging Multi-Modal Large Language Models for Autonomous Driving in Mines
With computer technology advancing in both software and hardware, the benefits of embodied intelligence are becoming increasingly evident. This robust interactive learning model enables artificial intelligence (AI) to be more flexibly deployed across diverse fields. In recent years, the development of multi-modal large language models (LLMs) has further accelerated the progress of AI, prompting extensive research on how to leverage these advancements to enhance the field of autonomous driving. This perspective believes that embodied intelligence can significantly enhance the application of LLMs, analyzing the new opportunities brought to the mining industry, and emphasizing the potential of their integration to revolutionize various aspects of the field. Meanwhile, This perspective also examines the challenges of deploying embodied agents in mining, while emphasizing their promising future and offering insights into potential research and development avenues.
- 29 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (4) , 4577-4578
Sora for Smart Mining: Towards Sustainability With Imaginative Intelligence and Parallel Intelligence
This letter summarizes discussions from IEEE TIV's Autonomous Mining Workshop, emphasizing the potential of video generation models in advancing smart mining.
- 29 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (4) , 4577-4578
Sora for Smart Mining: Towards Sustainability With Imaginative Intelligence and Parallel Intelligence
This letter summarizes discussions from IEEE TIV's Autonomous Mining Workshop, emphasizing the potential of video generation models in advancing smart mining.
- 26 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. PP (23798858) , 1-12
MiningLLM: Towards Mining 5.0 via Large Language Models in Autonomous Driving and Smart Mining
Minerals serve as the cornerstone of contemporary societal infrastructure and diverse industries. However, the mining sector perennially confronts transportation challenges marked by elevated expenses, diminished efficiency, and significant environmental ramifications. Although autonomous driving has been accomplished in open scenarios, its development in specialized areas remains limited. The open-pit mine, one of the typical unstructured environments, presents several challenges, encompassing complex geological conditions, and strict requirements for intelligent algorithms. These factors create formidable barriers to the implementation of autonomous driving technology in mining areas. The utilization of Large Language Models (LLMs) provides indispensable insights for addressing these challenges. This article introduces an assistant-driven approach via LLMs to realize Mining 5.0, representing an elevated level of intelligent mining operations. This exploration delves deeply into the necessity and feasibility of this approach from various perspectives, providing substantial theoretical support for the development of mining areas.
- 26 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. PP (23798858) , 1-12
MiningLLM: Towards Mining 5.0 via Large Language Models in Autonomous Driving and Smart Mining
Minerals serve as the cornerstone of contemporary societal infrastructure and diverse industries. However, the mining sector perennially confronts transportation challenges marked by elevated expenses, diminished efficiency, and significant environmental ramifications. Although autonomous driving has been accomplished in open scenarios, its development in specialized areas remains limited. The open-pit mine, one of the typical unstructured environments, presents several challenges, encompassing complex geological conditions, and strict requirements for intelligent algorithms. These factors create formidable barriers to the implementation of autonomous driving technology in mining areas. The utilization of Large Language Models (LLMs) provides indispensable insights for addressing these challenges. This article introduces an assistant-driven approach via LLMs to realize Mining 5.0, representing an elevated level of intelligent mining operations. This exploration delves deeply into the necessity and feasibility of this approach from various perspectives, providing substantial theoretical support for the development of mining areas.
- Open Access
- 8 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (3) , 4394-4404
Scenario Engineering for Autonomous Transportation: A New Stage in Open-Pit Mines
In recent years, open-pit mining has seen significant advancement, the cooperative operation of various specialized machinery substantially enhancing the efficiency of mineral extraction. However, the harsh environment and complex conditions in open-pit mines present substantial challenges for the implementation of autonomous transportation systems. This research introduces a novel paradigm that integrates Scenario Engineering (SE) with autonomous transportation systems to significantly improve the trustworthiness, robustness, and efficiency in open-pit mines by incorporating the four key components of SE, including Scenario Feature Extractor, Intelligence and Index, Calibration and Certification, and Verification and Validation. This paradigm has been validated in two famous open-pit mines, the experiment results demonstrate marked improvements in robustness, trustworthiness, and efficiency. By enhancing the capacity, scalability, and diversity of autonomous transportation, this paradigm fosters the integration of SE and parallel driving and finally propels the achievement of the ‘6S’ objectives.
- 8 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9 (3) , 4394-4404
Open AccessScenario Engineering for Autonomous Transportation: A New Stage in Open-Pit Mines
In recent years, open-pit mining has seen significant advancement, the cooperative operation of various specialized machinery substantially enhancing the efficiency of mineral extraction. However, the harsh environment and complex conditions in open-pit mines present substantial challenges for the implementation of autonomous transportation systems. This research introduces a novel paradigm that integrates Scenario Engineering (SE) with autonomous transportation systems to significantly improve the trustworthiness, robustness, and efficiency in open-pit mines by incorporating the four key components of SE, including Scenario Feature Extractor, Intelligence and Index, Calibration and Certification, and Verification and Validation. This paradigm has been validated in two famous open-pit mines, the experiment results demonstrate marked improvements in robustness, trustworthiness, and efficiency. By enhancing the capacity, scalability, and diversity of autonomous transportation, this paradigm fosters the integration of SE and parallel driving and finally propels the achievement of the ‘6S’ objectives.
- 20 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Radio Frequency Identification
- Vol. 8, 367-375
UnstrPrompt: Large Language Model Prompt for Driving in Unstructured Scenarios
The integration of language descriptions or prompts with Large Language Models (LLMs) into visual tasks is currently a focal point in the advancement of autonomous driving. This study has showcased notable advancements across various standard datasets. Nevertheless, the progress in integrating language prompts faces challenges in unstructured scenarios, primarily due to the limited availability of paired data. To address this challenge, we introduce a groundbreaking language prompt set called "UnstrPrompt." This prompt set is derived from three prominent unstructured autonomous driving datasets: IDD, ORFD, and AutoMine, collectively comprising a total of 6K language descriptions. In response to the distinctive features of unstructured scenarios, we have developed a structured approach for prompt generation, encompassing three key components: scene, road, and instance. Additionally, we provide a detailed overview of the language generation process and the validation procedures. We conduct tests on segmentation tasks, and our experiments have demonstrated that text-image fusion can improve accuracy by more than 3% on unstructured data. Additionally, our description architecture outperforms the generic urban architecture by more than 0.1%. This work holds the potential to advance various aspects such as interaction and foundational models in this scenario.
- 20 February 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Radio Frequency Identification
- Vol. 8, 367-375
UnstrPrompt: Large Language Model Prompt for Driving in Unstructured Scenarios
The integration of language descriptions or prompts with Large Language Models (LLMs) into visual tasks is currently a focal point in the advancement of autonomous driving. This study has showcased notable advancements across various standard datasets. Nevertheless, the progress in integrating language prompts faces challenges in unstructured scenarios, primarily due to the limited availability of paired data. To address this challenge, we introduce a groundbreaking language prompt set called "UnstrPrompt." This prompt set is derived from three prominent unstructured autonomous driving datasets: IDD, ORFD, and AutoMine, collectively comprising a total of 6K language descriptions. In response to the distinctive features of unstructured scenarios, we have developed a structured approach for prompt generation, encompassing three key components: scene, road, and instance. Additionally, we provide a detailed overview of the language generation process and the validation procedures. We conduct tests on segmentation tasks, and our experiments have demonstrated that text-image fusion can improve accuracy by more than 3% on unstructured data. Additionally, our description architecture outperforms the generic urban architecture by more than 0.1%. This work holds the potential to advance various aspects such as interaction and foundational models in this scenario.
- 25 July 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 8 (10) , 4395-4405
Semantic Loop Closure Detection for Intelligent Vehicles Using Panoramas
Loop Closure Detection is of great significance in the field of intelligent driving systems, as it reduces the cumulative error of the estimated position of the system and assists in generating a consistent global map. Existing methods differ in frame representation methods and the corresponding frame-matching strategy. Traditionally, local feature points and descriptors are studied extensively while recently global descriptors and semantic information extracted from deep learning methods are considered superior in terms of promoting a high-level understanding of the surrounding environments of robots. However, one of the most challenging problems of using semantic information for loop detection is how to deal with inconsistent visual contents from different viewpoints in the same place. In this paper, a semantic loop closure detection method using panoramas is proposed to address this issue. We design a pipeline for efficiently extracting and matching semantic information between frames to identify loops. Most importantly we propose a novel polar coordinate-based panorama representation to address the inconsistent visual appearance problem caused by viewpoint differences. Experiment results show that our proposed method can significantly increase the accuracy of loop closure detection tasks in challenging scenarios where traditional methods may fail.
- 25 July 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 8 (10) , 4395-4405
Semantic Loop Closure Detection for Intelligent Vehicles Using Panoramas
Loop Closure Detection is of great significance in the field of intelligent driving systems, as it reduces the cumulative error of the estimated position of the system and assists in generating a consistent global map. Existing methods differ in frame representation methods and the corresponding frame-matching strategy. Traditionally, local feature points and descriptors are studied extensively while recently global descriptors and semantic information extracted from deep learning methods are considered superior in terms of promoting a high-level understanding of the surrounding environments of robots. However, one of the most challenging problems of using semantic information for loop detection is how to deal with inconsistent visual contents from different viewpoints in the same place. In this paper, a semantic loop closure detection method using panoramas is proposed to address this issue. We design a pipeline for efficiently extracting and matching semantic information between frames to identify loops. Most importantly we propose a novel polar coordinate-based panorama representation to address the inconsistent visual appearance problem caused by viewpoint differences. Experiment results show that our proposed method can significantly increase the accuracy of loop closure detection tasks in challenging scenarios where traditional methods may fail.
- Top Cited Papers
- 9 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 8 (6) , 3692-3711
Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives
Intelligent vehicles (IVs) have gained worldwide attention due to their increased convenience, safety advantages, and potential commercial value. Despite predictions of commercial deployment by 2025, implementation remains limited to small-scale validation, with precise tracking controllers and motion planners being essential prerequisites for IVs. This paper reviews state-of-the-art motion planning methods for IVs, including pipeline planning and end-to-end planning methods. The study examines the selection, expansion, and optimization operations in a pipeline method, while it investigates training approaches and validation scenarios for driving tasks in end-to-end methods. Experimental platforms are reviewed to assist readers in choosing suitable training and validation strategies. A side-by-side comparison of the methods is provided to highlight their strengths and limitations, aiding system-level design choices. Current challenges and future perspectives are also discussed in this survey.
- 9 May 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 8 (6) , 3692-3711
Top Cited PapersMotion Planning for Autonomous Driving: The State of the Art and Future Perspectives
Intelligent vehicles (IVs) have gained worldwide attention due to their increased convenience, safety advantages, and potential commercial value. Despite predictions of commercial deployment by 2025, implementation remains limited to small-scale validation, with precise tracking controllers and motion planners being essential prerequisites for IVs. This paper reviews state-of-the-art motion planning methods for IVs, including pipeline planning and end-to-end planning methods. The study examines the selection, expansion, and optimization operations in a pipeline method, while it investigates training approaches and validation scenarios for driving tasks in end-to-end methods. Experimental platforms are reviewed to assist readers in choosing suitable training and validation strategies. A side-by-side comparison of the methods is provided to highlight their strengths and limitations, aiding system-level design choices. Current challenges and future perspectives are also discussed in this survey.
- Top Cited Papers
- 29 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 8 (1) , 673-683
Hierarchical Interpretable Imitation Learning for End-to-End Autonomous Driving
End-to-end autonomous driving provides a simple and efficient framework for autonomous driving systems, which can directly obtain control commands from raw perception data. However, it fails to address stability and interpretability problems in complex urban scenarios. In this paper, we construct a two-stage end-to-end autonomous driving model for complex urban scenarios, named HIIL (Hierarchical Interpretable Imitation Learning), which integrates interpretable BEV mask and steering angle to solve the problems shown above. In Stage One, we propose a pretrained Bird's Eye View (BEV) model which leverages a BEV mask to present an interpretation of the surrounding environment. In Stage Two, we construct an Interpretable Imitation Learning (IIL) model that fuses BEV latent feature from Stage One with an additional steering angle from Pure-Pursuit (PP) algorithm. In the HIIL model, visual information is converted to semantic images by the semantic segmentation network, and the semantic images are encoded to extract the BEV latent feature, which are decoded to predict BEV masks and fed to the IIL as perception data. In this way, the BEV latent feature bridges the BEV and IIL models. Visual information can be supplemented by the calculated steering angle for PP algorithm, speed vector, and location information, thus it could have better performance in complex and terrible scenarios. Our HIIL model meets an urgent requirement for interpretability and robustness of autonomous driving. We validate the proposed model in the CARLA simulator with extensive experiments which show remarkable interpretability, generalization, and robustness capability in unknown scenarios for navigation tasks.
- 29 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 8 (1) , 673-683
Top Cited PapersHierarchical Interpretable Imitation Learning for End-to-End Autonomous Driving
End-to-end autonomous driving provides a simple and efficient framework for autonomous driving systems, which can directly obtain control commands from raw perception data. However, it fails to address stability and interpretability problems in complex urban scenarios. In this paper, we construct a two-stage end-to-end autonomous driving model for complex urban scenarios, named HIIL (Hierarchical Interpretable Imitation Learning), which integrates interpretable BEV mask and steering angle to solve the problems shown above. In Stage One, we propose a pretrained Bird's Eye View (BEV) model which leverages a BEV mask to present an interpretation of the surrounding environment. In Stage Two, we construct an Interpretable Imitation Learning (IIL) model that fuses BEV latent feature from Stage One with an additional steering angle from Pure-Pursuit (PP) algorithm. In the HIIL model, visual information is converted to semantic images by the semantic segmentation network, and the semantic images are encoded to extract the BEV latent feature, which are decoded to predict BEV masks and fed to the IIL as perception data. In this way, the BEV latent feature bridges the BEV and IIL models. Visual information can be supplemented by the calculated steering angle for PP algorithm, speed vector, and location information, thus it could have better performance in complex and terrible scenarios. Our HIIL model meets an urgent requirement for interpretability and robustness of autonomous driving. We validate the proposed model in the CARLA simulator with extensive experiments which show remarkable interpretability, generalization, and robustness capability in unknown scenarios for navigation tasks.