3D Domain Adaptive Instance Segmentation via Cyclic Segmentation GANs
Open Access
- 30 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Biomedical and Health Informatics
- Vol. 27 (8) , 4018-4027
Abstract
3D instance segmentation for unlabeled imaging modalities is a challenging but essential task as collecting expert annotation can be expensive and time-consuming. Existing works segment a new modality by either deploying pre-trained models optimized on diverse training data or sequentially conducting image translation and segmentation with two relatively independent networks. In this work, we propose a novel Cyclic Segmentation Generative Adversarial Network ( CySGAN ) that conducts image translation and instance segmentation simultaneously using a unified network with weight sharing. Since the image translation layer can be removed at inference time, our proposed model does not introduce additional computational cost upon a standard segmentation model. For optimizing CySGAN, besides the CycleGAN losses for image translation and supervised losses for the annotated source domain, we also utilize self-supervised and segmentation-based adversarial objectives to enhance the model performance by leveraging unlabeled target domain images. We benchmark our approach on the task of 3D neuronal nuclei segmentation with annotated electron microscopy (EM) images and unlabeled expansion microscopy (ExM) data. The proposed CySGAN outperforms pre-trained generalist models, feature-level domain adaptation models, and the baselines that conduct image translation and segmentation sequentially. Our implementation and the newly collected, densely annotated ExM zebrafish brain nuclei dataset, named NucExM , are publicly available at https://connectomics-bazaar.github.io/proj/CySGAN/index.html .Keywords
Funding Information
- IFI program of the German Academic Exchange Service
- University of the Basque Country UPV/EHU (GIU19/027)
- Ministerio de Ciencia, Innovación y Universidades, AEI (PID2021-126701OB-I00)
- NIH (1R01EB024261)
- Lisa Yang, in part by John Doerr
- NIH (1R01MH123403)
- NIH (1R01MH123977)
- Schmidt Sciences
- N S F (IIS-1835231, IIS-2124179)
- NIH (5U54CA225088-03)
This publication has 20 references indexed in Scilit:
- Blastocyst-like structures generated solely from stem cellsNature, 2018
- Adversarial synthesis learning enables segmentation without target modality ground truth2018
- Least Squares Generative Adversarial Networks2017
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Adversarial Image Synthesis for Unpaired Multi-modal Cardiac Data2017
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Neuron Segmentation With High-Level Biological PriorsIEEE Transactions on Medical Imaging, 2017
- The Cityscapes Dataset for Semantic Urban Scene Understanding2016
- Expansion microscopyScience, 2015
- Watershed Cuts: Minimum Spanning Forests and the Drop of Water PrincipleIEEE Transactions on Pattern Analysis and Machine Intelligence, 2008