MDNet: Multimodal Cooperative Perception via Spatial Alignment of Modal Decision-Making
- 17 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 12 (11) , 16142-16154
Abstract
Through internet of things (IOT) communication technology, collaborative perception enhances a vehicle’s capacity to discern its surroundings while driving by integrating and synchronizing sensor data from multiple agents. With the advancement of cooperative perception techniques in single-modality methods, there has been a growing trend toward integrating multi-modal data from heterogeneous sensors in recent years. However, due to the data heterogeneity inherent in diverse sensors, Bird’s Eye View (BEV) maps generated from different types of sensors may exhibit local discrepancies in the spatial representation of entity positions. Furthermore, individual agents may produce uncertain and flawed feature representations in real noisy environments. The influence of this indeterminacy exacerbates the issue of local inconsistency, leading to misalignment of the detected target during BEV alignment and fusion, thereby reducing detection accuracy. To address these problems, we propose a modal decision-making spatial alignment cooperative perception network (MDNet). First, the network generates BEV feature maps through dense depth image supervision for voxel feature extraction and model-guided selective feature fusion. Subsequently, we achieve enhanced accuracy in object detection by performing spatial alignment of BEV representations generated from two distinct sensors, both globally and locally within the spatial domain. Besides, we employ a cascaded centralized pyramid strategy during the message fusion stage, facilitating flexible sampling across horizontal and vertical spatial dimensions, promoting deep interaction among multiple agents. We conduct quantitative and qualitative experiments on the public OPV2V and DAIR-V2X-C benchmarks, and our proposed MDNet exhibits superior performance and stronger robustness in the 3D object detection task, providing more precise target detection results.Keywords
Funding Information
- National Natural Science Foundation of China (62172441, 62172449)
- Joint Funds for Railway Fundamental Research of National Natural Science Foundation of China (U2368201)
- Special Fund of National Key Laboratory of Ni&Co Associated Minerals Resources Development and Comprehensive Utilization (GZSYS-KY-2022-018, GZSYS-KY-2022-024)
- Key Project of Shenzhen City Special Fund for Fundamental Research (202208183000751)
- National Natural Science Foundation of Hunan Province (2023JJ30696)
This publication has 28 references indexed in Scilit:
- V2VFusion: Multimodal Fusion for Enhanced Vehicle-to-Vehicle Cooperative Perception2023
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer2022
- Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D2020
- V2VNet: Vehicle-to-Vehicle Communication for Joint Perception and Prediction2020
- When2com: Multi-Agent Perception via Communication Graph Grouping2020
- PointPillars: Fast Encoders for Object Detection From Point Clouds2019
- VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection2018
- In Defense of Classical Image Processing: Fast Depth Completion on the CPU2018
- Deep Residual Learning for Image Recognition2016