Mask4Former: Mask Transformer for 4D Panoptic Segmentation
Authors:
- 13 May 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 9418-9425
Abstract
Accurately perceiving and tracking instances over time is essential for the decision-making processes of autonomous agents interacting safely in dynamic environments. With this intention, we propose Mask4Former for the challenging task of 4D panoptic segmentation of LiDAR point clouds. Mask4Former is the first transformer-based approach unifying semantic instance segmentation and tracking of sparse and irregular sequences of 3D point clouds into a single joint model. Our model directly predicts semantic instances and their temporal associations without relying on hand-crafted non-learned association strategies such as probabilistic clustering or voting-based center prediction. Instead, Mask4Former introduces spatio-temporal instance queries that encode the semantic and geometric properties of each semantic tracklet in the sequence. In an in-depth study, we find that promoting spatially compact instance predictions is critical as spatiotemporal instance queries tend to merge multiple semantically similar instances, even if they are spatially distant. To this end, we regress 6-DOF bounding box parameters from spatiotemporal instance queries, which are used as an auxiliary task to foster spatially compact predictions. Mask4Former achieves a new state-of-the-art on the SemanticKITTI test set with a score of 68.4 LSTQ.Keywords
All Related Versions
This publication has 43 references indexed in Scilit:
- 4D-StOP: Panoptic Segmentation of 4D LiDAR Using Spatio-Temporal Object Proposal Generation and Aggregation2023
- End-to-End Object Detection with Transformers2020
- Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution2020
- V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation2016
- Density-Based Clustering Based on Hierarchical Density Estimates2013
- Are we ready for autonomous driving? The KITTI vision benchmark suite2012
- Dimensionality Reduction by Learning an Invariant Mapping2006
- The Hungarian method for the assignment problemNaval Research Logistics (NRL), 2004
- A Boosted Particle Filter: Multitarget Detection and Tracking2004
- Mean shift: a robust approach toward feature space analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002