Context-Aware Transformer GAN for Direct Generation of Attenuation and Scatter Corrected PET Data
- 6 May 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Radiation and Plasma Medical Sciences
- Vol. 8 (6) , 677-689
Abstract
We present a context-aware generative deep learning framework to produce photon attenuation and scatter corrected (ASC) PET images directly from non-attenuation and non-scatter corrected (NASC) images. We trained conditional generative adversarial networks (cGAN) on either single-modality (NASC) or multi-modality (NASC+MRI) input data to map NASC images to pixel-wise continuously valued ASC PET images. We designed and evaluated four cGAN models including Pix2Pix, attention-guided cGAN (AG-Pix2Pix), vision transformer cGAN (ViT-GAN), and shifted window transformer cGAN (Swin-GAN). Retrospective 18F-fluorodeoxyglucose (18F-FDG) full-body PET images from 33 subjects were collected and analyzed. Notably, as a particular strength of this work, each patient in the study underwent both a PET/CT scan and a multisequence PET/MRI scan on the same day giving us a gold standard from the former as we investigate ASC for the latter. Quantitative analysis, evaluating image quality using peak signal-to-noise ratio (PSNR), multi-scale structural similarity index (MS-SSIM), normalized mean squared error (NRMSE), and mean absolute error (MAE) metrics, showed no significant impact of input type on PSNR (p=0.95), MS-SSIM (p=0.083), NRMSE (p=0.72), or MAE (p=0.70). For multi-modal input data, Swin-GAN outperformed Pix2Pix (p=0.023) and AG-Pix2Pix (p<0.001), but not ViT-GAN (p=0.154) in PSNR. Swin-GAN achieved significantly higher MS-SSIM than ViT-GAN (p=0.007) and AG-Pix2Pix (p=0.002). Multi-modal Swin-GAN demonstrated reduced NRMSE and MAE compared to ViT-GAN (p=0.023 and 0.031, respectively) and AG-Pix2Pix (both p<0.001), with marginal improvement over Pix2Pix (p<0.064). The cGAN models, in particular Swin-GAN, consistently generated reliable and accurate ASC PET images, whether using multi-modal or single-modal input data. The findings indicate this methodology can be used to generate ASC data from standalone PET scanners or integrated PET/MRI systems, without relying on transmission scan-based attenuation maps.Keywords
Funding Information
- NIH (R01EB03103801A1)
- Innovation Award from the Stanford Cancer Institute
- Finnish Cultural Foundation for his postdoctoral fellowship (00220062)
- Stanford NeuroTech Fellowship
- Stanford Graduate Fellowship
This publication has 57 references indexed in Scilit:
- Study of compatibility between a 3T MR system and detector modules for a second‐generation RF‐penetrable TOF‐PET insert for simultaneous PET/MRIMedical Physics, 2023
- Generalized image outpainting with U-transformerNeural Networks, 2023
- SwinGAN: A dual-domain Swin Transformer-based generative adversarial network for MRI reconstructionComputers in Biology and Medicine, 2022
- GANs for medical image analysisArtificial Intelligence in Medicine, 2020
- Generating synthetic CTs from magnetic resonance images using generative adversarial networksMedical Physics, 2018
- Attenuation Correction of PET/MR ImagingMagnetic Resonance Imaging Clinics of North America, 2017
- MR-based synthetic CT generation using a deep convolutional neural network methodMedical Physics, 2017
- Simultaneous PET/MR imaging with a radio frequency‐penetrable PET insertMedical Physics, 2016
- MRI for attenuation correction in PET: methods and challengesMagnetic Resonance Materials in Physics, Biology and Medicine, 2012
- Challenges and current methods for attenuation correction in PET/MRMagnetic Resonance Materials in Physics, Biology and Medicine, 2012