Multi-Sensor Learning Enables Information Transfer Across Different Sensory Data and Augments Multi-Modality Imaging
- 20 September 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (1) , 288-304
Abstract
Multi-modality imaging is widely used in clinical practice and biomedical research to gain a comprehensive understanding of an imaging subject. Currently, multi-modality imaging is accomplished by post hoc fusion of independently reconstructed images under the guidance of mutual information or spatially registered hardware, which limits the accuracy and utility of multi-modality imaging. Here, we investigate a data-driven multi-modality imaging (DMI) strategy for synergetic imaging of CT and MRI. We reveal two distinct types of features in multi-modality imaging, namely intra- and inter-modality features, and present a multi-sensor learning (MSL) framework to utilize the crossover inter-modality features for augmented multi-modality imaging. The MSL imaging approach breaks down the boundaries of traditional imaging modalities and allows for optimal hybridization of CT and MRI, which maximizes the use of sensory data. We showcase the effectiveness of our DMI strategy through synergetic CT-MRI brain imaging. The principle of DMI is quite general and holds enormous potential for various DMI applications across disciplines.Keywords
All Related Versions
Funding Information
- NIH (5 R01 CA256890, 1R01 CA256890617, 1R01CA227713)
- Research Grants Council of the Hong Kong Special Administrative Region (17308321, T45-401/22-N)
- Innovation and Technology Commission (ITS/273/22)
This publication has 64 references indexed in Scilit:
- AutoEncoder-Driven Multimodal Collaborative Learning for Medical Image SynthesisInternational Journal of Computer Vision, 2023
- Reconstruction of three‐dimensional tomographic patient models for radiation dose modulation in CT from two scout views using deep learningMedical Physics, 2022
- IREM: High-Resolution Magnetic Resonance Image Reconstruction via Implicit Neural Representation2021
- Cross-Modality Brain Structures Image Segmentation for the Radiotherapy Target Definition and Plan Optimization2021
- Modelling the Distribution of 3D Brain MRI Using a 2D Slice VAE2020
- Multimodal Unsupervised Image-to-Image Translation2018
- SimpleITK Image-Analysis Notebooks: a Collaborative Environment for Education and Reproducible ResearchJournal of Digital Imaging, 2017
- HybridMRI‐Ultrasound acquisitions, and scannerless real‐time imagingMagnetic Resonance in Medicine, 2016
- The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information RepositoryJournal of Digital Imaging, 2013
- A truly hybrid interventional MR/X-ray system: Feasibility demonstrationJournal of Magnetic Resonance Imaging, 2001