IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
Authors:
Top Cited Papers
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14905-14915
Abstract
Bird's eye view (BEV) representation has emerged as a dominant solution for describing 3D space in autonomous driving scenarios. However, objects in the BEV representation typically exhibit small sizes, and the associated point cloud context is inherently sparse, which leads to great challenges for reliable 3D perception. In this paper, we propose IS-Fusion, an innovative multimodal fusion framework that jointly captures the Instance- and Scene-level contextual information. IS-Fusion essentially differs from existing approaches that only focus on the BEV scene-level fusion by explicitly incorporating instance-level multimodal information, thus facilitating the instance-centric tasks like 3D object detection. It comprises a Hierarchical Scene Fusion (HSF) module and an Instance-Guided Fusion (IGF) module. HSF applies Point-to-Grid and Grid-to-Region transformers to capture the multimodal scene context at different granularities. IGF mines instance candidates, explores their relationships, and aggregates the local multimodal context for each instance. These instances then serve as guidance to enhance the scene feature and yield an instance-aware BEV representation. On the challenging nuScenes benchmark, IS-Fusion outperforms all the published multimodal works to date. Code is available at: https://github.com/yinjunbo/IS-Fusion.Keywords
All Related Versions
Funding Information
- FDCT (0154/2022/A3,0102/2023/RIA2,001/2024/SKL,MYRG-CRG2022-00013-IOTSC-ICI,SRG2022-00023-IOTSC)
- National Natural Science Foundation of China (62372405)
This publication has 71 references indexed in Scilit:
- Graph R-CNN: Towards Accurate 3D Object Detection with Semantic-Decorated Local Graph2022
- Deformable Feature Aggregation for Dynamic Multi-modal 3D Object Detection2022
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- PillarNet: Real-Time and High-Performance Pillar-Based 3D Object Detection2022
- PETR: Position Embedding Transformation for Multi-view 3D Object Detection2022
- BRNet: Exploring Comprehensive Features for Monocular Depth Estimation2022
- Semi-supervised 3D Object Detection with Proficient Teachers2022
- Deep Continuous Fusion for Multi-sensor 3D Object Detection2018
- Deep Residual Learning for Image Recognition2016
- The Hungarian method for the assignment problemNaval Research Logistics Quarterly, 1955