CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes
Top Cited Papers
- 30 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 10 (4) , 3134-3141
Abstract
Leveraging multiple sensors is crucial for robust semantic perception in autonomous driving, as each sensor type has complementary strengths and weaknesses. However, existing sensor fusion methods often treat sensors uniformly across all conditions, leading to suboptimal performance. By contrast, we propose a novel, condition-aware multimodal fusion approach for robust semantic perception of driving scenes. Our method, CAFuser, uses an RGB camera input to classify environmental conditions and generate a Condition Token that guides the fusion of multiple sensor modalities. We further newly introduce modality-specific feature adapters to align diverse sensor inputs into a shared latent space, enabling efficient integration with a single and shared pre-trained backbone. By dynamically adapting sensor fusion based on the actual condition, our model significantly improves robustness and accuracy, especially in adverse-condition scenarios. CAFuser ranks first on the public MUSES benchmarks, achieving 59.7 PQ for multimodal panoptic and 78.2 mIoU for semantic segmentation, and also sets the new state of the art on DeLiVER.Keywords
Funding Information
- ETH Future Computing Laboratory
- donation from Huawei Technologies
This publication has 29 references indexed in Scilit:
- SAMFusion: Sensor-Adaptive Multimodal Fusion for 3D Object Detection in Adverse Weather2024
- MUSES: The Multi-sensor Semantic Perception Dataset for Driving Under Uncertainty2024
- Learning Modality-Agnostic Representation for Semantic Segmentation from Any Modalities2024
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Semantic scene segmentation for robotics2022
- EfficientPS: Efficient Panoptic SegmentationInternational Journal of Computer Vision, 2021
- Panoptic Segmentation2019
- Deep Continuous Fusion for Multi-sensor 3D Object Detection2018
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012