MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection
Preprint
Abstract
Monocular 3D object detection has long been a challenging task in autonomous driving. Most existing methods follow conventional 2D detectors to first localize object centers, and then predict 3D attributes by neighboring features. However, only using local visual features is insufficient to understand the scene-level 3D spatial structures and ignores the long-range inter-object depth relations. In this paper, we introduce the first DETR framework for Monocular DEtection with a depth-guided TRansformer, named MonoDETR. We modify the vanilla transformer to be depth-aware and guide the whole detection process by contextual depth cues. Specifically, concurrent to the visual encoder that captures object appearances, we introduce to predict a foreground depth map, and specialize a depth encoder to extract non-local depth embeddings. Then, we formulate 3D object candidates as learnable queries and propose a depth-guided decoder to conduct object-scene depth interactions. In this way, each object query estimates its 3D attributes adaptively from the depth-guided regions on the image and is no longer constrained to local visual features. On KITTI benchmark with monocular images as input, MonoDETR achieves state-of-the-art performance and requires no extra dense depth annotations. Besides, our depth-guided modules can also be plug-and-play to enhance multi-view 3D object detectors on nuScenes dataset, demonstrating our superior generalization capacity. Code is available at https://github.com/ZrrSkywalker/MonoDETR.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images2023
- PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection2023
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- PETR: Position Embedding Transformation for Multi-view 3D Object Detection2022
- MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformer2022
- AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection2021
- Conditional DETR for Fast Training Convergence2021
- Categorical Depth Distribution Network for Monocular 3D Object Detection2021
- nuScenes: A Multimodal Dataset for Autonomous Driving2020
- SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation2020