Adversarial Semantic Hallucination for Domain Generalized Semantic Segmentation
Authors:
- 1 January 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3849-3858
Abstract
Convolutional neural networks typically perform poorly when the test (target domain) and training (source domain) data have significantly different distributions. While this problem can be mitigated by using the target domain data to align the source and target domain feature representations, the target domain data may be unavailable due to privacy concerns. Consequently, there is a need for methods that generalize well despite restricted access to target domain data during training. In this work, we propose an adversarial semantic hallucination approach (ASH), which combines a class-conditioned hallucination module and a semantic segmentation module. Since the segmentation performance varies across different classes, we design a semantic-conditioned style hallucination module to generate affine transformation parameters from semantic information in the segmentation probability maps of the source domain image. Unlike previous adaptation approaches, which treat all classes equally, ASH considers the class-wise differences. The segmentation module and the hallucination module compete adversarially, with the hallucination module generating increasingly "difficult" stylized images to challenge the segmentation module. In response, the segmentation module improves as it is trained with generated samples at an appropriate class-wise difficulty level. Our results on the Cityscapes and Mapillary benchmark datasets show that our method is competitive with state of the art work. Code is made available at https://github.com/gabriel-tjio/ASH.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization2017
- The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes2017
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Domain-Adversarial Training of Neural Networks2017
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks2017
- DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes2016
- Deep Residual Learning for Image Recognition2016
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015