Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN
Preprint
Abstract
Generative Adversarial Networks (GAN) have many potential medical imaging applications, including data augmentation, domain adaptation, and model explanation. Due to the limited memory of Graphical Processing Units (GPUs), most current 3D GAN models are trained on low-resolution medical images, these models either cannot scale to high-resolution or are prone to patchy artifacts. In this work, we propose a novel end-to-end GAN architecture that can generate high-resolution 3D images. We achieve this goal by separating training and inference. During training, we adopt a hierarchical structure that simultaneously generates a low-resolution version of the image and a randomly selected sub-volume of the high-resolution image. The hierarchical design has two advantages: First, the memory demand for training on high-resolution images is amortized among subvolumes. Furthermore, anchoring the high-resolution subvolumes to a single low-resolution image ensures anatomical consistency between subvolumes. During inference, our model can directly generate full high-resolution images. We also incorporate an encoder with a similar hierarchical structure into the model to extract features from the images. Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation. We also demonstrate clinical applications of the proposed model in data augmentation, image super-resolution and clinical-relevant feature extraction.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Vox2Vox: 3D-GAN for Brain Tumour Segmentation2021
- FastSurfer - A fast and accurate deep learning based neuroimaging pipelineNeuroImage, 2020
- MedGAN: Medical image translation using GANsComputerized Medical Imaging and Graphics, 2020
- GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classificationNeurocomputing, 2018
- Generating Highly Realistic Images of Skin Lesions with GANs2018
- Super-resolution CT Image Reconstruction Based on Dictionary Learning and Sparse RepresentationScientific Reports, 2018
- Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic LossIEEE Transactions on Medical Imaging, 2018
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measuresScientific Data, 2015
- Optimal threshold in CT quantification of emphysemaEuropean Radiology, 2012