Enhancing Generative Models for Modality Imputation of 3-D MRIs via Consistency-Aware Refinement and Super-Resolution Guidance
- 1 November 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 37 (11) , 6329-6343
Abstract
Reconstructing missing modalities of magnetic resonance images (MRIs) is a significant challenge in the medical imaging field. Current generative approaches such as generative adversarial networks (GANs) and diffusion models (DFs) have shown promise in synthesizing high-fidelity 2-D slices. However, they fall short in producing high-quality 3-D results due to their inability to leverage the contextual information across adjacent slices, resulting in low-quality images with poor interplane consistency. This problem is exacerbated when trained on datasets obtained with 2-D scanning protocols, where different MRI modalities have varying resolution between slices, leading to blurry and low-resolution results. To overcome these issues, we propose a novel fine-tuning strategy that can enhance the 2-D multimodal synthesis models to improve both consistency and interplane resolution in 3-D images. We begin by developing a novel attention-based module that can effectively empower generative models to produce 3-D results with high consistency between adjacent slices. Furthermore, we incorporate self-supervised super-resolution (SR) to deal with the varying resolution problem and eventually improve the interplane resolution of generated images. Instead of directly applying SR models that may bring about potential artifacts, we design a novel uncertainty modulation strategy, which optimizes the utilization of super-resolved images for high-quality reconstruction. We extensively evaluate our method through supervised and unsupervised multimodal synthesis using different generative models, including GAN- and DM-based models, all of which demonstrate the superiority and flexibility of the proposed method.Keywords
Funding Information
- National Natural Science Foundation of China (62471288)
- Fundamental Research Funds for the Central Universities (YG2023LC07)
This publication has 73 references indexed in Scilit:
- Cross-conditioned Diffusion Model for Medical Image to Image Translation2024
- Arbitrary Reduction of MRI Inter-slice Spacing Using Hierarchical Feature Conditional Diffusion2023
- Alias-Free Co-modulated Network for Cross-Modality Synthesis and Super-Resolution of MR Images2023
- Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis2023
- InverseSR: 3D Brain MRI Super-Resolution Using a Latent Diffusion Model2023
- CoLa-Diff: Conditional Latent Diffusion Model for Multi-modal MRI Synthesis2023
- AutoGAN-Synthesizer: Neural Architecture Search for Cross-Modality MRI Synthesis2022
- DiamondGAN: Unified Multi-modal Generative Adversarial Networks for MRI Sequences Synthesis2019
- Learning Blind Video Temporal Consistency2018
- Super‐resolution musculoskeletal MRI using deep learningMagnetic Resonance in Medicine, 2018