Bidirectional feature matching based on deep pairwise contrastive learning for multiparametric MRI image synthesis
Open Access
- 15 June 2023
- journal article
- research article
- Published by IOP Publishing in Physics in Medicine & Biology
- Vol. 68 (12) , 125010
Abstract
Objective. Multi-parametric MR image synthesis is an effective approach for several clinical applications where specific modalities may be unavailable to reach a diagnosis. While technical and practical conditions limit the acquisition of new modalities for a patient, multimodal image synthesis combines multiple modalities to synthesize the desired modality. Approach. In this paper, we propose a new multi-parametric magnetic resonance imaging (MRI) synthesis model, which generates the target MRI modality from two other available modalities, in pathological MR images. We first adopt a contrastive learning approach that trains an encoder network to extract a suitable feature representation of the target space. Secondly, we build a synthesis network that generates the target image from a common feature space that approximately matches the contrastive learned space of the target modality. We incorporate a bidirectional feature learning strategy that learns a multimodal feature matching function, in two opposite directions, to transform the augmented multichannel input in the learned target space. Overall, our training synthesis loss is expressed as the combination of the reconstruction loss and a bidirectional triplet loss, using a pair of features. Main results. Compared to other state-of-the-art methods, the proposed model achieved an average improvement rate of 3.9% and 3.6% on the IXI and BraTS’18 datasets respectively. On the tumor BraTS’18 dataset, our model records the highest Dice score of 0.793(0.04) for preserving the synthesized tumor regions in the segmented images. Significance. Validation of the proposed model on two public datasets confirms the efficiency of the model to generate different MR contrasts, and preserve tumor areas in the synthesized images. In addition, the model is flexible to generate head and neck CT image from MR acquisitions. In future work, we plan to validate the model using interventional iMRI contrasts for MR-guided neurosurgery applications, and also for radiotherapy applications. Clinical measurements will be collected during surgery to evaluate the model’s performance.Keywords
Funding Information
- Canadian Network for Research and Innovation in Machining Technology, Natural Sciences and Engineering Research Council of Canada
This publication has 18 references indexed in Scilit:
- Ea-GANs: Edge-Aware Generative Adversarial Networks for Cross-Modality MR Image SynthesisIEEE Transactions on Medical Imaging, 2019
- Image Synthesis in Multi-Contrast MRI With Conditional Generative Adversarial NetworksIEEE Transactions on Medical Imaging, 2019
- Multimodal MR Synthesis via Modality-Invariant Latent RepresentationIEEE Transactions on Medical Imaging, 2017
- Robust Multi-modal MR Image Synthesis2017
- Fully automatic, multiorgan segmentation in normal whole body magnetic resonance imaging (MRI), using classification forests (CFs), convolutional neural networks (CNNs), and a multi‐atlas (MA) approachMedical Physics, 2017
- The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)IEEE Transactions on Medical Imaging, 2015
- Tumors as Organs: Complex Tissues that Interface with the Entire OrganismDevelopmental Cell, 2010
- Multicomponent water proton transverse relaxation and T2-discriminated water diffusion in myelinated and nonmyelinated nerveMagnetic Resonance Imaging, 1998
- Use of proton magnetic resonance spectroscopy for monitoring disease progression in multiple sclerosisAnnals of Neurology, 1994
- The Divergence and Bhattacharyya Distance Measures in Signal SelectionIEEE Transactions on Communications, 1967