A Camera-Based End-to-End Autonomous Driving Framework Combined With Meta-Based Multi-Task Optimization
- 17 September 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Transportation Electrification
- Vol. 11 (1) , 4443-4455
Abstract
Most existing autonomous driving pipelines can be divided into two broad categories: those based on a modular framework, which can result in error transmission, and those based on an end-to-end framework, which lack interpretability. To overcome these challenges, we propose a novel vision-based multi-task framework that incorporates motion planning, Bird’s Eye View(BEV) map generation, BEV object prediction, depth estimation, semantic segmentation, and velocity prediction. In particular, we present an improved view transformation module that transforms feature maps into BEV space and predicts future waypoints in BEV space. The multi-task framework can improve performance by sharing information across tasks, and the results of the multi-tasks also improve interpretability. In addition, to address the negative transfer, we introduce inter-task affinity, which provides a rough estimate of the relationship between tasks. Moreover, because these relationships may change during training, we use a meta-based multi-task optimization method to dynamically adjust the multi-task weighting. We evaluate the performance of our proposed model using the Longest6 and Town05 Long benchmarks of the CARLA simulator. Our model outperforms the current state-of-the-art camera-based models and achieves competitive results with other multi-modal methods on both benchmarks. These results demonstrate the considerable potential of our proposed model for autonomous driving systems. we have also prepared an autonomous driving demonstration using the CARLA simulator which is presented at https://www.youtube.com/watch?v=ctngFH4GSBc.Keywords
Funding Information
- National Natural Science Foundation of China (52225212, U20A20333, U20A20331, 52072160, 52272418, U22A20100)
- National Key Research and Development Program of China (2022YFB2503302)
- Key Project for the Development of Strategic Emerging Industries of Jiangsu Province (BE2020083-3, BE2020083-2)
This publication has 45 references indexed in Scilit:
- 3D Object Detection for Autonomous Driving: A Comprehensive SurveyInternational Journal of Computer Vision, 2023
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- PETR: Position Embedding Transformation for Multi-view 3D Object Detection2022
- ST-P3: End-to-End Vision-Based Autonomous Driving via Spatial-Temporal Feature Learning2022
- A study of the effects of negative transfer on deep unsupervised domain adaptation methodsExpert Systems with Applications, 2021
- Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D2020
- A negative transfer approach to person re-identification via domain augmentationInformation Sciences, 2020
- DSDNet: Deep Structured Self-driving Network2020
- Gradient-based optimizer: A new metaheuristic optimization algorithmInformation Sciences, 2020
- Gradient-Based Multiobjective Optimization with Uncertainties2017