UniFusion: Unified Multi-view Fusion Transformer for Spatial-Temporal Representation in Bird’s-Eye-View
Authors:
- Jingyu ChenJCJingyu Chen
- Chao ChenCCChao Chen
- Xiaozhi ChenXCXiaozhi Chen
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8656-8665
Abstract
Bird’s eye view (BEV) representation is a new perception formulation for autonomous driving, which is based on spatial fusion. Further, temporal fusion is also introduced in BEV representation and gains great success. In this work, we propose a new method that unifies both spatial and temporal fusion and merges them into a unified mathematical formulation. The unified fusion could not only provide a new perspective on BEV fusion but also brings new capabilities. With the proposed unified spatial-temporal fusion, our method could support long-range fusion, which is hard to achieve in conventional BEV methods. Moreover, the BEV fusion in our work is temporal-adaptive and the weights of temporal fusion are learnable. In contrast, conventional methods mainly use fixed and equal weights for temporal fusion. Besides, the proposed unified fusion could avoid information lost in conventional BEV fusion methods and make full use of features. Extensive experiments and ablation studies on the NuScenes dataset show the effectiveness of the proposed method and our method gains the state-of-the-art performance in the map and vehicle segmentation task.Keywords
All Related Versions
Funding Information
- Research and Development
- National Natural Science Foundation of China
- National Science Foundation
- Research and Development
This publication has 20 references indexed in Scilit:
- BEVFormer: Learning Bird’s-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- Is Pseudo-Lidar needed for Monocular 3D Object detection?2021
- Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D2020
- nuScenes: A Multimodal Dataset for Autonomous Driving2020
- An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection2019
- Deformable Convolutional Networks2017
- Deep Residual Learning for Image Recognition2016
- ImageNet: A large-scale hierarchical image database2009
- Stereo vision for planetary rovers: Stochastic modeling to near real-time implementationInternational Journal of Computer Vision, 1992