Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN
Preprint
Abstract
Generative Adversarial Networks (GAN) have many potential medical imaging applications, including data augmentation, domain adaptation, and model explanation. Due to the limited memory of Graphical Processing Units (GPUs), most current 3D GAN models are trained on low-resolution medical images, these models either cannot scale to high-resolution or are prone to patchy artifacts. In this work, we propose a novel end-to-end GAN architecture that can generate high-resolution 3D images. We achieve this goal by using different configurations between training and inference. During training, we adopt a hierarchical structure that simultaneously generates a low-resolution version of the image and a randomly selected sub-volume of the high-resolution image. The hierarchical design has two advantages: First, the memory demand for training on high-resolution images is amortized among sub-volumes. Furthermore, anchoring the high-resolution sub-volumes to a single low-resolution image ensures anatomical consistency between sub-volumes. During inference, our model can directly generate full high-resolution images. We also incorporate an encoder with a similar hierarchical structure into the model to extract features from the images. Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation. We also demonstrate clinical applications of the proposed model in data augmentation and clinical-relevant feature extraction.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Generating 3D TOF-MRA volumes and segmentation labels using generative adversarial networksMedical Image Analysis, 2022
- Vox2Vox: 3D-GAN for Brain Tumour Segmentation2021
- MedGAN: Medical image translation using GANsComputerized Medical Imaging and Graphics, 2020
- GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classificationNeurocomputing, 2018
- Improving Spatial Resolution at CT: Development, Benefits, and PitfallsRadiology, 2018
- 3-D Convolutional Encoder-Decoder Network for Low-Dose CT via Transfer Learning From a 2-D Trained NetworkIEEE Transactions on Medical Imaging, 2018
- Compressed Sensing MRI Reconstruction Using a Generative Adversarial Network With a Cyclic LossIEEE Transactions on Medical Imaging, 2018
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measuresScientific Data, 2015
- Accuracy of high-resolution CT in diagnosing lung diseases.American Journal of Roentgenology, 1998