MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection
Authors:
Top Cited Papers
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9121-9132
Abstract
Monocular 3D object detection has long been a challenging task in autonomous driving. Most existing methods follow conventional 2D detectors to first localize object centers, and then predict 3D attributes by neighboring features. However, only using local visual features is insufficient to understand the scene-level 3D spatial structures and ignores the long-range inter-object depth relations. In this paper, we introduce the first DETR framework for Monocular DEtection with a depth-guided TRansformer, named MonoDETR. We modify the vanilla transformer to be depth-aware and guide the whole detection process by contextual depth cues. Specifically, concurrent to the visual encoder that captures object appearances, we introduce to predict a foreground depth map, and specialize a depth encoder to extract non-local depth embeddings. Then, we formulate 3D object candidates as learnable queries and propose a depth-guided decoder to conduct object-scene depth interactions. In this way, each object query estimates its 3D attributes adaptively from the depth-guided regions on the image and is no longer constrained to local visual features. On KITTI benchmark with monocular images as input, MonoDETR achieves state-of-the-art performance and requires no extra dense depth annotations. Besides, our depth-guided modules can also be plug-and-play to enhance multi-view 3D object detectors on nuScenes dataset, demonstrating our superior generalization capacity. Code is available at https://github.com/ZrrSkywalker/MonoDETR.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
- Innovation and Technology Commission
This publication has 39 references indexed in Scilit:
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- PETR: Position Embedding Transformation for Multi-view 3D Object Detection2022
- IAFA: Instance-Aware Feature Aggregation for 3D Object Detection from a Single Image2021
- RTM3D: Real-Time Monocular 3D Detection from Object Keypoints for Autonomous Driving2020
- Rethinking Pseudo-LiDAR Representation2020
- Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D2020
- Kinematic 3D Object Detection in Monocular Video2020
- End-to-End Object Detection with Transformers2020
- Monocular 3D Object Detection for Autonomous Driving2016
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012