T-MAE: Temporal Masked Autoencoders for Point Cloud Representation Learning
Authors:
- Theo GeversTGTheo Gevers
- Martin R. OswaldMOMartin R. Oswald
Preprint
Abstract
The scarcity of annotated data in outdoor point cloud segmentation poses a significant obstacle in harnessing the modeling capabilities of advanced networks like transformers. Consequently, scholars have been actively investigating efficacious self-supervised pre-training strategies, e.g. contrasting learning and reconstruction-based pretext tasks. Nevertheless, temporal information, which is inherent in the LiDAR point cloud sequence, is consistently disregarded. To better utilize this property, we propose an effective pre-training strategy, namely Temporal Masked AutoEncoders (T-MAE), which takes as input temporally adjacent frames and learns temporal dependency. A SiamWCA backbone, containing a Siamese encoder and a window-based cross-attention (WCA) module, is established for the two-frame input. Taking into account that the motion of an ego-vehicle alters the illumination angles of the same instance, temporal modeling also serves as a robust and natural data augmentation, enhancing the comprehension of target objects. Moreover, instead of utilizing consecutive frames, it is more cost-effective and powerful by using distant historical frames. SiamWCA is a powerful architecture but heavily relies on annotated data. With our T-MAE pre-training strategy, we achieve the best performance on the Waymo dataset among self-supervised learning methods. Comprehensive experiments are conducted to validate all components of our proposal. Upon acceptance, the source code will be made accessible.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- Masked Autoencoders for Point Cloud Self-supervised Learning2022
- Masked Discrimination for Self-supervised Learning on Point Clouds2022
- SWFormer: Sparse Window Transformer for 3D Object Detection in Point Clouds2022
- A Closer Look at Invariances in Self-supervised Pre-training for 3D Vision2022
- ProposalContrast: Unsupervised Pre-training for LiDAR-Based 3D Object Detection2022
- Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts2021
- PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding2020
- End-to-End Object Detection with Transformers2020
- nuScenes: A Multimodal Dataset for Autonomous Driving2020
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset2020