Is image-to-image translation the panacea for multimodal image registration? A comparative study
Open Access
- 28 November 2022
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 17 (11) , e0276196
Abstract
Despite current advancement in the field of biomedical image processing, propelled by the deep learning revolution, multimodal image registration, due to its several challenges, is still often performed manually by specialists. The recent success of image-to-image (I2I) translation in computer vision applications and its growing use in biomedical areas provide a tempting possibility of transforming the multimodal registration problem into a, potentially easier, monomodal one. We conduct an empirical study of the applicability of modern I2I translation methods for the task of rigid registration of multimodal biomedical and medical 2D and 3D images. We compare the performance of four Generative Adversarial Network (GAN)-based I2I translation methods and one contrastive representation learning method, subsequently combined with two representative monomodal registration methods, to judge the effectiveness of modality translation for multimodal image registration. We evaluate these method combinations on four publicly available multimodal (2D and 3D) datasets and compare with the performance of registration achieved by several well-known approaches acting directly on multimodal image data. Our results suggest that, although I2I translation may be helpful when the modalities to register are clearly correlated, registration of modalities which express distinctly different properties of the sample are not well handled by the I2I translation approach. The evaluated representation learning method, which aims to find abstract image-like representations of the information shared between the modalities, manages better, and so does the Mutual Information maximisation approach, acting directly on the original multimodal images. We share our complete experimental setup as open-source (https://github.com/MIDA-group/MultiRegEval), including method implementations, evaluation code, and all datasets, for further reproducing and benchmarking.Keywords
All Related Versions
Funding Information
- Wallenberg AI, Autonomous Systems and Software Program (WASP) AI-Math initiative
- VINNOVA (2017-02447)
- Vetenskapsrådet (2017-04385)
This publication has 75 references indexed in Scilit:
- MIND: Modality independent neighbourhood descriptor for multi-modal deformable registrationMedical Image Analysis, 2012
- Adaptive Stochastic Gradient Descent Optimisation for Image RegistrationInternational Journal of Computer Vision, 2008
- Medical image registrationPhysics in Medicine & Biology, 2001
- Image registration by maximization of combined mutual information and gradient informationIEEE Transactions on Medical Imaging, 2000
- An overlap invariant entropy measure of 3D medical image alignmentPattern Recognition, 1999
- Comparison and Evaluation of Retrospective Intermodality Brain Image Registration TechniquesJournal of Computer Assisted Tomography, 1997
- Multimodality image registration by maximization of mutual informationIEEE Transactions on Medical Imaging, 1997
- Multi-modal volume registration by maximization of mutual informationMedical Image Analysis, 1996
- A survey of image registration techniquesACM Computing Surveys, 1992
- Random sample consensusCommunications of the ACM, 1981