Tuopu Wen
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Tuopu Wen has a total of 47 co-authors. Here are the top co-authors by number of shared publications:
- Kun Jiang
Tsinghua University
16 shared publications - Diange Yang
Tsinghua University
15 shared publications - Mengmeng Yang
Tsinghua University
9 shared publications - Zheng Fu
School of Vehicle and Mobility, Tsinghua University, Beijing, China
7 shared publications - Benny Wijaya
School of Vehicle and Mobility, Tsinghua University, Beijing, China
6 shared publications - Jinyu Miao
Tsinghua University
5 shared publications
Latest Publications
- 11 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 27  (5) , 5072-5086
Temporal Range-Point-Voxel Fusion for Unified BEV Scene Perception and Motion Prediction
LiDAR-based bird’s-eye-view (BEV) perception has emerged as an appealing approach for practical autonomous driving applications due to its direct leveraging of precise 3D structures and delivering efficient performance. This paradigm aims to jointly determine the semantics and motion states of various traffic participants on BEV grids. However, most existing LiDAR-based BEV perception methods primarily focus on motion prediction, leading to inferior semantic performance. To address this limitation, we propose a novel multi-frame, multi-view, and multi-task unified framework in this work, which enhances scene perception for both improved BEV semantic segmentation and comparative motion prediction performances. Our framework, named temporal range-point-voxel fusion (T-RPVFusion), leverages a sequence of LiDAR sweeps as input and jointly outputs semantic and motion information on BEV grids. In T-RPVFusion, we first introduce a novel multi-view semantic encoder that extracts high-quality semantic features from each LiDAR sweep. These semantic feature maps are then aggregated into an integrated feature map using the proposed bi-layer spatio-temporal pyramid network. Subsequently, the integrated feature map undergoes processing in both the semantic and motion heads and yields corresponding outputs, respectively. Extensive experiments conducted on Waymo and nuScenes show that our method outperforms previous state-of-the-art (SOTA) in terms of BEV semantic segmentation, while concurrently demonstrating comparable performance in motion prediction. Notably, our method achieves a significant improvement on BEV semantic segmentation task, attaining a mIOU of 49.5%, surpassing the previous SOTA with a great margin of + 12.1% mIOU on Waymo Open Dataset. The code is available at https://github.com/thuwyl/trpvfusion
- 11 March 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 27  (5) , 5072-5086
Temporal Range-Point-Voxel Fusion for Unified BEV Scene Perception and Motion Prediction
LiDAR-based bird’s-eye-view (BEV) perception has emerged as an appealing approach for practical autonomous driving applications due to its direct leveraging of precise 3D structures and delivering efficient performance. This paradigm aims to jointly determine the semantics and motion states of various traffic participants on BEV grids. However, most existing LiDAR-based BEV perception methods primarily focus on motion prediction, leading to inferior semantic performance. To address this limitation, we propose a novel multi-frame, multi-view, and multi-task unified framework in this work, which enhances scene perception for both improved BEV semantic segmentation and comparative motion prediction performances. Our framework, named temporal range-point-voxel fusion (T-RPVFusion), leverages a sequence of LiDAR sweeps as input and jointly outputs semantic and motion information on BEV grids. In T-RPVFusion, we first introduce a novel multi-view semantic encoder that extracts high-quality semantic features from each LiDAR sweep. These semantic feature maps are then aggregated into an integrated feature map using the proposed bi-layer spatio-temporal pyramid network. Subsequently, the integrated feature map undergoes processing in both the semantic and motion heads and yields corresponding outputs, respectively. Extensive experiments conducted on Waymo and nuScenes show that our method outperforms previous state-of-the-art (SOTA) in terms of BEV semantic segmentation, while concurrently demonstrating comparable performance in motion prediction. Notably, our method achieves a significant improvement on BEV semantic segmentation task, attaining a mIOU of 49.5%, surpassing the previous SOTA with a great margin of + 12.1% mIOU on Waymo Open Dataset. The code is available at https://github.com/thuwyl/trpvfusion
- 10 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 26  (12) , 22538-22555
Toward Democratizing High-Definition Map Update Through Consortium Blockchain
In the rapidly evolving landscape of autonomous vehicles and advanced navigation systems, the accuracy of high-definition maps and real-time updating has become paramount. However, in this progression, the security of map data has not received adequate attention, although the accuracy of the data can be easily altered when the system is breached. Thus, this paper introduces a novel approach to democratizing the process of high-definition map updates by leveraging consortium blockchain technology specifically designed for Proof of Presence and Reputation (POP-R) to safeguard the update process. Our proposed system leverages the presence and reputation of vehicles through infrastructure nodes to enhance the accuracy and reliability of HD map updates. We created a trusted ecosystem for maintaining high-definition maps, marked by a superior safety score across three scenarios compared to the standard proof of reputation technique. Additionally, it demonstrates high efficiency, achieving 12,000 transactions per second (TPS) for data queries and more than 2,500 TPS for data writing in our blockchain network. This efficiency proved our prowess in the lightweight computational power required, suitable for decentralized and crowdsourced-based systems. Through our POP-R framework, we lay the foundation for a new decentralized approach to the evolution of high-definition maps in the era of autonomous mobility.
- 10 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 26  (12) , 22538-22555
Toward Democratizing High-Definition Map Update Through Consortium Blockchain
In the rapidly evolving landscape of autonomous vehicles and advanced navigation systems, the accuracy of high-definition maps and real-time updating has become paramount. However, in this progression, the security of map data has not received adequate attention, although the accuracy of the data can be easily altered when the system is breached. Thus, this paper introduces a novel approach to democratizing the process of high-definition map updates by leveraging consortium blockchain technology specifically designed for Proof of Presence and Reputation (POP-R) to safeguard the update process. Our proposed system leverages the presence and reputation of vehicles through infrastructure nodes to enhance the accuracy and reliability of HD map updates. We created a trusted ecosystem for maintaining high-definition maps, marked by a superior safety score across three scenarios compared to the standard proof of reputation technique. Additionally, it demonstrates high efficiency, achieving 12,000 transactions per second (TPS) for data queries and more than 2,500 TPS for data writing in our blockchain network. This efficiency proved our prowess in the lightweight computational power required, suitable for decentralized and crowdsourced-based systems. Through our POP-R framework, we lay the foundation for a new decentralized approach to the evolution of high-definition maps in the era of autonomous mobility.
- 9 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10  (11) , 12213-12220
C2F-Planner: Interaction-Aware Coarse-to-Fine Planning for Autonomous Vehicles
Ensuring safe and socially compliant driving is essential for autonomous vehicle planning. However, one of the significant challenges remains the performance bottleneck caused by interaction uncertainty in complex traffic scenarios. Traditional planning algorithms typically account for all traffic participants, leading to overly generalized predictions and decisions that do not adequately capture essential interactive dynamics. In real-world driving, autonomous vehicles interact primarily with a subset of surrounding agents, whose behaviors are dynamically influenced by the vehicle's decisions. To address this, we propose a novel framework with two key innovations. First, we introduce a supervised learning-based approach that simplifies the complex interaction agents recognition into a classification problem, incorporating a dynamically weighted attention mechanism to identify potential interaction agents accurately. Secondly, we develop a coarse-to-fine decision decoding strategy. Initially, the autonomous driving system makes preliminary coarse decisions based on the scene context. These decisions are then integrated with interaction agents to form multiple cohesive interaction units. Within these units, fine-grained interaction reasoning is employed to predict the future behaviors of interaction agents and refine the ego vehicle's decisions accordingly. Experimental results indicate that our method significantly enhances the accuracy of interaction modelling and the robustness of decision-making in complex scenarios, providing a scalable solution to improve the safety and intelligence of autonomous driving systems. Code available at: https://github.com/JaneFo/C2F-Planner.git
- 9 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10  (11) , 12213-12220
C2F-Planner: Interaction-Aware Coarse-to-Fine Planning for Autonomous Vehicles
Ensuring safe and socially compliant driving is essential for autonomous vehicle planning. However, one of the significant challenges remains the performance bottleneck caused by interaction uncertainty in complex traffic scenarios. Traditional planning algorithms typically account for all traffic participants, leading to overly generalized predictions and decisions that do not adequately capture essential interactive dynamics. In real-world driving, autonomous vehicles interact primarily with a subset of surrounding agents, whose behaviors are dynamically influenced by the vehicle's decisions. To address this, we propose a novel framework with two key innovations. First, we introduce a supervised learning-based approach that simplifies the complex interaction agents recognition into a classification problem, incorporating a dynamically weighted attention mechanism to identify potential interaction agents accurately. Secondly, we develop a coarse-to-fine decision decoding strategy. Initially, the autonomous driving system makes preliminary coarse decisions based on the scene context. These decisions are then integrated with interaction agents to form multiple cohesive interaction units. Within these units, fine-grained interaction reasoning is employed to predict the future behaviors of interaction agents and refine the ego vehicle's decisions accordingly. Experimental results indicate that our method significantly enhances the accuracy of interaction modelling and the robustness of decision-making in complex scenarios, providing a scalable solution to improve the safety and intelligence of autonomous driving systems. Code available at: https://github.com/JaneFo/C2F-Planner.git
- 2 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. PP, 1-8
Residual Learning Towards High-Fidelity Vehicle Dynamics Modeling With Transformer
The vehicle dynamics model serves as a vital component of autonomous driving systems, as it describes the temporal changes in vehicle state. Traditional physics-based methods employ mathematical formulae to model vehicle dynamics, but they are unable to adequately describe complex vehicle systems due to the simplifications they entail. The deep learning-based methods directly regress vehicle dynamics, but their performance and generalization capabilities still require further enhancement. In this letter, we address these problems by proposing a vehicle dynamics correction system that leverages deep neural networks to correct the state residuals of a physical model instead of directly estimating the states. This system greatly reduces the difficulty of network learning and thus improves the estimation accuracy of vehicle dynamics. Furthermore, we have developed a novel Transformer-based dynamics residual correction network, DyTR. This network implicitly represents state residuals as high-dimensional queries, and iteratively updates the estimated residuals by interacting with dynamics state features. Simulation experiments and real-world scaled-vehicle testing demonstrate the proposed vehicle dynamics correction system works much better than the physics-based vehicle model, and our proposed DyTR model achieves the state-of-the-art performance.
- 2 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. PP, 1-8
Residual Learning Towards High-Fidelity Vehicle Dynamics Modeling With Transformer
The vehicle dynamics model serves as a vital component of autonomous driving systems, as it describes the temporal changes in vehicle state. Traditional physics-based methods employ mathematical formulae to model vehicle dynamics, but they are unable to adequately describe complex vehicle systems due to the simplifications they entail. The deep learning-based methods directly regress vehicle dynamics, but their performance and generalization capabilities still require further enhancement. In this letter, we address these problems by proposing a vehicle dynamics correction system that leverages deep neural networks to correct the state residuals of a physical model instead of directly estimating the states. This system greatly reduces the difficulty of network learning and thus improves the estimation accuracy of vehicle dynamics. Furthermore, we have developed a novel Transformer-based dynamics residual correction network, DyTR. This network implicitly represents state residuals as high-dimensional queries, and iteratively updates the estimated residuals by interacting with dynamics state features. Simulation experiments and real-world scaled-vehicle testing demonstrate the proposed vehicle dynamics correction system works much better than the physics-based vehicle model, and our proposed DyTR model achieves the state-of-the-art performance.
- 7 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10  (6) , 5417-5424
Enhancing Lane Segment Perception and Topology Reasoning With Crowdsourcing Trajectory Priors
In autonomous driving, recent advances in online mapping provide autonomous vehicles with a comprehensive understanding of driving scenarios. Moreover, incorporating prior information input into such perception model represents an effective approach to ensure the robustness and accuracy. However, utilizing diverse sources of prior information still faces three key challenges: the acquisition of high-quality prior information, alignment between prior and online perception, efficient integration. To address these issues, we investigate prior augmentation from a novel perspective of trajectory priors. In this paper, we initially extract crowdsourcing trajectory data from Argoverse2 motion forecasting dataset and encode trajectory data into rasterized heatmap and vectorized instance tokens, then we incorporate such prior information into the online mapping model through different ways. Besides, with the purpose of mitigating the misalignment between prior and online perception, we design a confidence-based fusion module that takes alignment into account during the fusion process. We conduct extensive experiments on OpenLane-V2 dataset. The results indicate that our method's performance significantly outperforms the current state-of-the-art methods. Code is released is at https://github.com/wowlza/TrajTopo
- 7 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10  (6) , 5417-5424
Enhancing Lane Segment Perception and Topology Reasoning With Crowdsourcing Trajectory Priors
In autonomous driving, recent advances in online mapping provide autonomous vehicles with a comprehensive understanding of driving scenarios. Moreover, incorporating prior information input into such perception model represents an effective approach to ensure the robustness and accuracy. However, utilizing diverse sources of prior information still faces three key challenges: the acquisition of high-quality prior information, alignment between prior and online perception, efficient integration. To address these issues, we investigate prior augmentation from a novel perspective of trajectory priors. In this paper, we initially extract crowdsourcing trajectory data from Argoverse2 motion forecasting dataset and encode trajectory data into rasterized heatmap and vectorized instance tokens, then we incorporate such prior information into the online mapping model through different ways. Besides, with the purpose of mitigating the misalignment between prior and online perception, we design a confidence-based fusion module that takes alignment into account during the fusion process. We conduct extensive experiments on OpenLane-V2 dataset. The results indicate that our method's performance significantly outperforms the current state-of-the-art methods. Code is released is at https://github.com/wowlza/TrajTopo
- 11 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 26  (2) , 2212-2226
Top-Down Attention-Based Mechanisms for Interpretable Autonomous Driving
Despite the remarkable advancements in autonomous driving, the challenge persists in achieving interpretable action decision-making, primarily owing to the intricate and ambiguous relationship between detected agents and driving intention. In this study, we introduce an interpretable action prediction model, denoted as the Prediction-Driven Attention Network (PDANet), designed to undertake action decisions and provide corresponding interpretations cohesively. The PDANet is inspired by the perceptual mechanisms inherent in human drivers, who allocate attention according to their driving intentions. Specifically, we elaborate a prediction module to generate vehicle prospective trajectories to characterize driving intentions. Subsequently, the features of this predicted trajectory are utilized to modulate the attention distribution among agents through the top-down attention module, yielding an attention map. Finally, two distinct task tokens are applied to aggregate agent features and generate the final output according to the derived attention map. Extensive experiments conducted on the publicly available BDD-OIA and nu-AR datasets demonstrate that our proposed method outperforms all prior works in terms of both action prediction and behavior interpretation tasks. Remarkably, our method attains a noteworthy enhancement in the behavior interpretation task, surpassing the previous state-of-the-art by a substantial margin of +10.8% in terms of F1-score on the nu-AR dataset. We also validate our algorithm on Carla Town05 long in a closed-loop decision-making scenario, highlighting the generality and robustness of our approach. Furthermore, qualitative results show that the agents selected by our model are more closely aligned with human cognitive processes.
- 11 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Transportation Systems
- Vol. 26  (2) , 2212-2226
Top-Down Attention-Based Mechanisms for Interpretable Autonomous Driving
Despite the remarkable advancements in autonomous driving, the challenge persists in achieving interpretable action decision-making, primarily owing to the intricate and ambiguous relationship between detected agents and driving intention. In this study, we introduce an interpretable action prediction model, denoted as the Prediction-Driven Attention Network (PDANet), designed to undertake action decisions and provide corresponding interpretations cohesively. The PDANet is inspired by the perceptual mechanisms inherent in human drivers, who allocate attention according to their driving intentions. Specifically, we elaborate a prediction module to generate vehicle prospective trajectories to characterize driving intentions. Subsequently, the features of this predicted trajectory are utilized to modulate the attention distribution among agents through the top-down attention module, yielding an attention map. Finally, two distinct task tokens are applied to aggregate agent features and generate the final output according to the derived attention map. Extensive experiments conducted on the publicly available BDD-OIA and nu-AR datasets demonstrate that our proposed method outperforms all prior works in terms of both action prediction and behavior interpretation tasks. Remarkably, our method attains a noteworthy enhancement in the behavior interpretation task, surpassing the previous state-of-the-art by a substantial margin of +10.8% in terms of F1-score on the nu-AR dataset. We also validate our algorithm on Carla Town05 long in a closed-loop decision-making scenario, highlighting the generality and robustness of our approach. Furthermore, qualitative results show that the agents selected by our model are more closely aligned with human cognitive processes.
- 6 September 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9  (11) , 9836-9843
DiffMap: Enhancing Map Segmentation With Map Prior Using Diffusion Model
Constructing high-definition (HD) maps is a crucial requirement for enabling autonomous driving. In recent years, several map segmentation algorithms have been developed to address this need, leveraging advancements in Bird's-Eye View (BEV) perception. However, existing models still encounter challenges in producing realistic and consistent semantic map layouts. A prominent issue is the limited utilization of structured priors inherent in map segmentation masks. In light of this, we propose DiffMap, a novel approach specifically designed to model the structured priors of map segmentation masks using latent diffusion model. By incorporating this technique, the performance of existing semantic segmentation methods can be significantly enhanced and certain structural errors present in the segmentation outputs can be effectively rectified. Notably, the proposed module can be seamlessly integrated into any map segmentation model, thereby augmenting its capability to accurately delineate semantic information. Furthermore, through extensive visualization analysis, our model demonstrates superior proficiency in generating results that more accurately reflect real-world map layouts, further validating its efficacy in improving the quality of the generated maps.
- 6 September 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9  (11) , 9836-9843
DiffMap: Enhancing Map Segmentation With Map Prior Using Diffusion Model
Constructing high-definition (HD) maps is a crucial requirement for enabling autonomous driving. In recent years, several map segmentation algorithms have been developed to address this need, leveraging advancements in Bird's-Eye View (BEV) perception. However, existing models still encounter challenges in producing realistic and consistent semantic map layouts. A prominent issue is the limited utilization of structured priors inherent in map segmentation masks. In light of this, we propose DiffMap, a novel approach specifically designed to model the structured priors of map segmentation masks using latent diffusion model. By incorporating this technique, the performance of existing semantic segmentation methods can be significantly enhanced and certain structural errors present in the segmentation outputs can be effectively rectified. Notably, the proposed module can be seamlessly integrated into any map segmentation model, thereby augmenting its capability to accurately delineate semantic information. Furthermore, through extensive visualization analysis, our model demonstrates superior proficiency in generating results that more accurately reflect real-world map layouts, further validating its efficacy in improving the quality of the generated maps.
- 19 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 10  (4) , 2519-2550
A Survey on Monocular Re-Localization: From the Perspective of Scene Map Representation
Monocular Re-Localization (MRL) is an essential component in autonomous applications, estimating 6 degree-of-freedom ego poses w.r.t. the scene map using monocular images. Over the past few decades, there has been notable advancement in the field of MRL techniques. Several works have achieved remarkable success in terms of both the accuracy and reliability of localization. In MRL, scene maps are represented in diverse forms, and they dictate the functionality and efficacy of MRL methods. However, to the best of our knowledge, existing surveys do not provide systematic reviews about the correlation between MRL solutions and their used scene map representation. This survey fills the gap by comprehensively reviewing MRL methods from such a perspective, promoting further research. 1) We commence by delving into the problem definition of MRL, exploring current challenges, and comparing ours with existing surveys. 2) MRL methods are categorized and reviewed into five classes based on the representation forms of the utilized map, i.e., geo-tagged frames, visual landmarks, point clouds, vectorized semantic maps, and neural network-based maps. 3) To quantitatively and fairly compare MRL methods using different maps, we introduce some public datasets and evaluate the performances of several advanced MRL approaches. The strengths and shortcomings of MRL methods using various maps are analyzed. 4) Finally, we introduce some topics of interest in this field and provide our individual viewpoints. This survey can serve as a valuable resource for MRL, and a summary of this survey is accessible to the community at: github.com/jinyummiao/map-in-mono-reloc
- 19 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 10  (4) , 2519-2550
A Survey on Monocular Re-Localization: From the Perspective of Scene Map Representation
Monocular Re-Localization (MRL) is an essential component in autonomous applications, estimating 6 degree-of-freedom ego poses w.r.t. the scene map using monocular images. Over the past few decades, there has been notable advancement in the field of MRL techniques. Several works have achieved remarkable success in terms of both the accuracy and reliability of localization. In MRL, scene maps are represented in diverse forms, and they dictate the functionality and efficacy of MRL methods. However, to the best of our knowledge, existing surveys do not provide systematic reviews about the correlation between MRL solutions and their used scene map representation. This survey fills the gap by comprehensively reviewing MRL methods from such a perspective, promoting further research. 1) We commence by delving into the problem definition of MRL, exploring current challenges, and comparing ours with existing surveys. 2) MRL methods are categorized and reviewed into five classes based on the representation forms of the utilized map, i.e., geo-tagged frames, visual landmarks, point clouds, vectorized semantic maps, and neural network-based maps. 3) To quantitatively and fairly compare MRL methods using different maps, we introduce some public datasets and evaluate the performances of several advanced MRL approaches. The strengths and shortcomings of MRL methods using various maps are analyzed. 4) Finally, we introduce some topics of interest in this field and provide our individual viewpoints. This survey can serve as a valuable resource for MRL, and a summary of this survey is accessible to the community at: github.com/jinyummiao/map-in-mono-reloc
- 30 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9  (10) , 5973-5994
High-Definition Maps Construction Based on Visual Sensor: A Comprehensive Survey
In recent years, the field of autonomous vehicles has seen a significant increase in academic research, with high-definition (HD) maps emerging as a critical component of autonomous driving technology. These maps provide intricate details of road networks and serve as a fundamental input for critical tasks such as vehicle positioning, navigation and decision making. Given the widespread availability and affordability of camera sensors, they have become an indispensable aspect of autonomous vehicles, justifying their focus in this review. The aim of this article is to provide researchers with a comprehensive overview and summary of recent advances in HD mapping. The review begins with a concise summary of the key frameworks and background information relevant to the creation of HD maps using camera sensors. This is followed by a comprehensive review of the research methods used for map production, including both offline and online approaches. In particular, network-based mapping methods have emerged as a prominent area of research in the field of HD mapping. In response to this burgeoning trend, this article presents a comprehensive and meticulous overview of the diverse research efforts in this particular field. Finally, the article addresses pertinent issues and future challenges with the aim of guiding researchers in understanding the current trends and methodologies prevalent in the field.
- 30 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Intelligent Vehicles
- Vol. 9  (10) , 5973-5994
High-Definition Maps Construction Based on Visual Sensor: A Comprehensive Survey
In recent years, the field of autonomous vehicles has seen a significant increase in academic research, with high-definition (HD) maps emerging as a critical component of autonomous driving technology. These maps provide intricate details of road networks and serve as a fundamental input for critical tasks such as vehicle positioning, navigation and decision making. Given the widespread availability and affordability of camera sensors, they have become an indispensable aspect of autonomous vehicles, justifying their focus in this review. The aim of this article is to provide researchers with a comprehensive overview and summary of recent advances in HD mapping. The review begins with a concise summary of the key frameworks and background information relevant to the creation of HD maps using camera sensors. This is followed by a comprehensive review of the research methods used for map production, including both offline and online approaches. In particular, network-based mapping methods have emerged as a prominent area of research in the field of HD mapping. In response to this burgeoning trend, this article presents a comprehensive and meticulous overview of the diverse research efforts in this particular field. Finally, the article addresses pertinent issues and future challenges with the aim of guiding researchers in understanding the current trends and methodologies prevalent in the field.
- 29 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9  (1) , 803-810
Poses as Queries: End-to-End Image-to-LiDAR Map Localization With Transformers
High-precision vehicle localization with commercial setups is a crucial technique for high-level autonomous driving tasks. As a newly emerged approach, monocular localization in LiDAR map achieves promising balance between cost and accuracy, but estimating pose by finding correspondences between such cross-modal sensor data is challenging, thereby damaging the localization accuracy. In this letter, we address the problem by proposing a novel Transformer-based neural network to register 2D images into 3D LiDAR map in an end-to-end manner. We first implicitly represent poses as high-dimensional feature vectors called pose queries and gradually optimize poses by interacting with the retrieved relevant information from cross-modal features using attention mechanism in a proposed POse Estimator Transformer (POET) module. Moreover, we apply a multiple hypotheses aggregation method that estimates the final poses by performing parallel optimization on multiple randomly initialized pose queries to reduce the network uncertainty. Comprehensive analysis and experimental results on public benchmarks conclude that the proposed image-to-LiDAR map localization network could achieve state-of-the-art performances in challenging cross-modal localization tasks.
- 29 November 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9  (1) , 803-810
Poses as Queries: End-to-End Image-to-LiDAR Map Localization With Transformers
High-precision vehicle localization with commercial setups is a crucial technique for high-level autonomous driving tasks. As a newly emerged approach, monocular localization in LiDAR map achieves promising balance between cost and accuracy, but estimating pose by finding correspondences between such cross-modal sensor data is challenging, thereby damaging the localization accuracy. In this letter, we address the problem by proposing a novel Transformer-based neural network to register 2D images into 3D LiDAR map in an end-to-end manner. We first implicitly represent poses as high-dimensional feature vectors called pose queries and gradually optimize poses by interacting with the retrieved relevant information from cross-modal features using attention mechanism in a proposed POse Estimator Transformer (POET) module. Moreover, we apply a multiple hypotheses aggregation method that estimates the final poses by performing parallel optimization on multiple randomly initialized pose queries to reduce the network uncertainty. Comprehensive analysis and experimental results on public benchmarks conclude that the proposed image-to-LiDAR map localization network could achieve state-of-the-art performances in challenging cross-modal localization tasks.