Multimodal Transformer for Accelerated MR Imaging
Top Cited Papers
- 15 June 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 42 (10) , 2804-2816
Abstract
Accelerated multi-modal magnetic resonance (MR) imaging is a new and effective solution for fast MR imaging, providing superior performance in restoring the target modality from its undersampled counterpart with guidance from an auxiliary modality. However, existing works simply combine the auxiliary modality as prior information, lacking in-depth investigations on the potential mechanisms for fusing different modalities. Further, they usually rely on the convolutional neural networks (CNNs), which is limited by the intrinsic locality in capturing the long-distance dependency. To this end, we propose a multi-modal transformer (MTrans), which is capable of transferring multi-scale features from the target modality to the auxiliary modality, for accelerated MR imaging. To capture deep multi-modal information, our MTrans utilizes an improved multi-head attention mechanism, named cross attention module, which absorbs features from the auxiliary modality that contribute to the target modality. Our framework provides three appealing benefits: (i) Our MTrans use an improved transformers for multi-modal MR imaging, affording more global information compared with existing CNN-based methods. (ii) A new cross attention module is proposed to exploit the useful information in each modality at different scales. The small patch in the target modality aims to keep more fine details, the large patch in the auxiliary modality aims to obtain high-level context features from the larger region and supplement the target modality effectively. (iii) We evaluate MTrans with various accelerated multi-modal MR imaging tasks, e.g., MR image reconstruction and super-resolution, where MTrans outperforms state-of-the-art methods on fastMRI and real-world clinical datasets.Keywords
All Related Versions
Funding Information
- National Key Research and Development Program of China (2018AAA0100100)
- Advanced Manufacturing and Engineering (AME) Programmatic Fund (A20H4b0141)
This publication has 63 references indexed in Scilit:
- Accelerated High-Dimensional MR Imaging With Sparse Sampling Using Low-Rank TensorsIEEE Transactions on Medical Imaging, 2016
- Exponential Wavelet Iterative Shrinkage Thresholding Algorithm for compressed sensing magnetic resonance imagingInformation Sciences, 2015
- P‐LORAKS: Low‐rank modeling of local k‐space neighborhoods with parallel imaging dataMagnetic Resonance in Medicine, 2015
- Accuracy of 3‐T MRI using susceptibility‐weighted imaging to detect meniscal tears of the kneeKnee Surgery, Sports Traumatology, Arthroscopy, 2014
- Super-resolution reconstruction of cardiac MRI using coupled dictionary learning2014
- PROMISE: Parallel‐imaging and compressed‐sensing reconstruction of multicontrast imaging using SharablE informationMagnetic Resonance in Medicine, 2014
- Low-Rank Modeling of Local $k$-Space Neighborhoods (LORAKS) for Constrained MRIIEEE Transactions on Medical Imaging, 2013
- ESPIRiT—an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPAMagnetic Resonance in Medicine, 2013
- Multi‐contrast reconstruction with Bayesian compressed sensingMagnetic Resonance in Medicine, 2011
- SPIRiT: Iterative self‐consistent parallel imaging reconstruction from arbitrary k‐spaceMagnetic Resonance in Medicine, 2010