Generic-to-Personalised Learning for Multimodal Image Synthesis With Bidirectional Variational GAN
- 14 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 28 (15209210) , 902-914
Abstract
Multimodal image synthesis, which predicts target-modality images from source-modality images, has garnered considerable attention in the field of clinical diagnosis. Both unidirectional and bidirectional multimodal image synthesis methods have been explored in the medical domain, however, unidirectional models heavily rely on paired images, while current bidirectional models typically overlook local image details due to their unsupervised training patterns. In this work, we propose a Bidirectional Variational Generative Adversarial Network (BVGAN) for multimodal image synthesis, which achieves high-quality bidirectional translations between any two modalities using only a limited number paired images. Firstly, BVGAN's generator incorporates a variational structure (VAS) to regularise the latent space for noise reduction. This regularisation imposes smoothness to the latent space, enabling BVGAN to produce high-quality, noise-free images. Secondly, a novel generic-to-personalised (GTP) learning strategy is introduced to train BVGAN and reduce its reliance on a large sets of paired images. GTP initially leverages an unsupervised learning model to capture the global mapping between two modalities using unpaired images from generic patients. It then applies a supervised learning model to refine the mapping for individual patient, enhancing image details. Finally, the GTP learning strategy along with VAS enables BVGAN to achieve state-of-the-art performance on two multi-modality medical datasets: Brain CTMRI and BRATS.Keywords
Funding Information
- National Key R&D Program of China (2024YFA1012700)
- National Natural Science Foundation of China (62476035)
- National Natural Science Foundation of China (62206037)
- National Natural Science Foundation of China (U24B20140)
- Young Elite Scientists Sponsorship Program by CAST (2024QNRC001)
This publication has 39 references indexed in Scilit:
- Cross-conditioned Diffusion Model for Medical Image to Image Translation2024
- Cross-Modality Synthesis of T1c MRI from Non-contrast Images Using GANs: Implications for Brain Tumor Research2024
- Bi-MGAN: Bidirectional T1-to-T2 MRI images prediction using multi-generative multi-adversarial netsBiomedical Signal Processing and Control, 2022
- Brain MR to PET Synthesis via Bidirectional Generative Adversarial Network2020
- Deep learning approaches using 2D and 3D convolutional neural networks for generating male pelvic synthetic computed tomography from magnetic resonance imagingMedical Physics, 2019
- Unpaired Brain MR-to-CT Synthesis Using a Structure-Constrained CycleGAN2018
- MR-based synthetic CT generation using a deep convolutional neural network methodMedical Physics, 2017
- Estimating CT Image from MRI Data Using 3D Fully Convolutional Networks2016
- Patch Based Synthesis of Whole Head MR Images: Application To EPI Distortion Correction2016
- A Compressed Sensing Approach for MR Tissue Contrast Synthesis2011