MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16907-16916
Abstract
Test-time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi-modal extension of test-time adaptation for 3D semantic segmentation. We find that, directly applying existing methods usually results in performance instability at test time, because multi-modal input is not considered jointly. To design a framework that can take full advantage of multi-modality, where each modality provides regularized self-supervisory signals to other modalities, we propose two complementary modules within and across the modalities. First, Intra-modal Pseudo-label Generation (Intra-PG) is introduced to obtain reliable pseudo labels within each modality by aggregating information from two models that are both pre-trained on source data but updated with target data at different paces. Second, Inter-modal Pseudo-label Refinement (Inter-PR) adaptively selects more reliable pseudo labels from different modalities based on a proposed consistency scheme. Experiments demonstrate that our regularized pseudo labels produce stable self-learning signals in numerous multi-modal test-time adaptation scenarios for 3D semantic segmentation. Visit our project website at https://www.nec-labs.com/~mas/MM-TTAKeywords
All Related Versions
Funding Information
- National Research Foundation of Korea (NRF-2020M3H8A1115028,FY2021)
This publication has 27 references indexed in Scilit:
- RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving2019
- Cross-Sensor Deep Domain Adaptation for LiDAR Detection and Segmentation2019
- SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud2019
- Unsupervised Domain Adaptation for Semantic Segmentation via Class-Balanced Self-training2018
- 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks2018
- Learning to Adapt Structured Output Space for Semantic Segmentation2018
- SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud2018
- The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes2016
- Deep Residual Learning for Image Recognition2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015