Mx2M: Masked Cross-Modality Modeling in Domain Adaptation for 3D Semantic Segmentation
Authors:
- Boxiang ZhangBZBoxiang Zhang
- Zunran WangZWZunran Wang
- Yuanyuan GuanYGYuanyuan Guan
- Wenhui LiWLWenhui Li
Preprint
Abstract
Existing methods of cross-modal domain adaptation for 3D semantic segmentation predict results only via 2D-3D complementarity that is obtained by cross-modal feature matching. However, as lacking supervision in the target domain, the complementarity is not always reliable. The results are not ideal when the domain gap is large. To solve the problem of lacking supervision, we introduce masked modeling into this task and propose a method Mx2M, which utilizes masked cross-modality modeling to reduce the large domain gap. Our Mx2M contains two components. One is the core solution, cross-modal removal and prediction (xMRP), which makes the Mx2M adapt to various scenarios and provides cross-modal self-supervision. The other is a new way of cross-modal feature matching, the dynamic cross-modal filter (DxMF) that ensures the whole method dynamically uses more suitable 2D-3D complementarity. Evaluation of the Mx2M on three DA scenarios, including Day/Night, USA/Singapore, and A2D2/SemanticKITTI, brings large improvements over previous methods on many metrics.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Learning 3D Semantic Segmentation with only 2D Image Supervision2021
- Adversarial unsupervised domain adaptation for 3D semantic segmentation with multi-modal learningISPRS Journal of Photogrammetry and Remote Sensing, 2021
- SOLO: Segmenting Objects by Locations2020
- Unsupervised scene adaptation for semantic segmentation of urban mobile laser scanning point cloudsISPRS Journal of Photogrammetry and Remote Sensing, 2020
- MFENet: Multi-level feature enhancement network for real-time semantic segmentationNeurocomputing, 2020
- 3DMV: Joint 3D-Multi-view Prediction for 3D Semantic Scene Segmentation2018
- Learning to Adapt Structured Output Space for Semantic Segmentation2018
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Deep Residual Learning for Image Recognition2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015