Influence of early through late fusion on pancreas segmentation from imperfectly registered multimodal magnetic resonance imaging
Authors:
Open Access
- 1 March 2025
- journal article
- Published by SPIE-Intl Soc Optical Eng in Journal of Medical Imaging
- Vol. 12 (02) , 1-12
Abstract
PurposeCombining different types of medical imaging data, through multimodal fusion, promises better segmentation of anatomical structures, such as the pancreas. Strategic implementation of multimodal fusion could improve our ability to study diseases such as diabetes. However, where to perform fusion in deep learning models is still an open question. It is unclear if there is a single best location to fuse information when analyzing pairs of imperfectly aligned images or if the optimal fusion location depends on the specific model being used. Two main challenges when using multiple imaging modalities to study the pancreas are (1) the pancreas and surrounding abdominal anatomy have a deformable structure, making it difficult to consistently align the images and (2) breathing by the individual during image collection further complicates the alignment between multimodal images. Even after using state-of-the-art deformable image registration techniques, specifically designed to align abdominal images, multimodal images of the abdomen are often not perfectly aligned. We examine how the choice of different fusion points, ranging from early in the image processing pipeline to later stages, impacts the segmentation of the pancreas on imperfectly registered multimodal magnetic resonance (MR) images.ApproachOur dataset consists of 353 pairs of T2-weighted (T2w) and T1-weighted (T1w) abdominal MR images from 163 subjects with accompanying pancreas segmentation labels drawn mainly based on the T2w images. Because the T2w images were acquired in an interleaved manner across two breath holds and the T1w images on one breath hold, there were three different breath holds impacting the alignment of each pair of images. We used deeds, a state-of-the-art deformable abdominal image registration method to align the image pairs. Then, we trained a collection of basic UNets with different fusion points, spanning from early to late layers in the model, to assess how early through late fusion influenced segmentation performance on imperfectly aligned images. To investigate whether performance differences on key fusion points are generalized to other architectures, we expanded our experiments to nnUNet.ResultsThe single-modality T2w baseline using a basic UNet model had a median Dice score of 0.766, whereas the same baseline on the nnUNet model achieved 0.824. For each fusion approach, we analyzed the differences in performance with Dice residuals, by subtracting the baseline score from the fusion score for each datapoint. For the basic UNet, the best fusion approach was from early/mid fusion and occurred in the middle of the encoder with a median Dice residual of +0.012 compared with the baseline. For the nnUNet, the best fusion approach was early fusion through naïve image concatenation before the model, with a median Dice residual of +0.004 compared with the baseline. After Bonferroni correction, the distributions of the Dice scores for these best fusion approaches were found to be statistically significant (p<0.05) via the paired Wilcoxon signed-rank test against the baseline.ConclusionsFusion in specific blocks can improve performance, but the best blocks for fusion are model-specific, and the gains are small. In imperfectly registered datasets, fusion is a nuanced problem, with the art of design remaining vital for uncovering potential insights. Future innovation is needed to better address fusion in cases of imperfect alignment of abdominal image pairs. The code associated with this project is available here https://github.com/MASILab/influence_of_fusion_on_pancreas_segmentation.Keywords
Funding Information
- National Institutes of Health
- Vanderbilt Institute for Clinical and Translational Research, Vanderbilt University Medical Center
- Integrated Training in Engineering and Diabetes (T32 DK101003)
- National Science Foundation Graduate Research Fellowship (DGE-1746891)
- Leona M. and Harry B. Helmsley Charitable Trust (2207-05374)
- JDRF International (3-SRA-2015-102-M-B)
- National Institute of Diabetes and Digestive and Kidney Diseases (DK 129979)
- National Center for Advancing Translational Sciences (UL1 TR000445)
- Vanderbilt Diabetes Research and Training Center, Division of Diabetes, Endocrinology, and Metabolic Diseases (DK 020593)
- Vanderbilt University Institute of Imaging Science Center for Human Imaging (1 S10OD021771 01)
- NIH (R01DK135597)
- National Science Foundation (1452485)
- National Center for Advancing Translational Sciences (NCATS) Clinical Translational Science Award (CTSA) Program (5UL1TR002243-03)
This publication has 27 references indexed in Scilit:
- Pancreas Segmentation in MRI Using Graph-Based Decision Fusion on Convolutional Neural Networks2016
- Evaluation of Six Registration Methods for the Human Abdomen on Clinically Acquired CTIEEE Transactions on Biomedical Engineering, 2016
- The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)IEEE Transactions on Medical Imaging, 2015
- Altered Volume, Morphology and Composition of the Pancreas in Type 2 DiabetesPLOS ONE, 2015
- Robust abdominal imaging with incomplete breath-holdsMagnetic Resonance in Medicine, 2013
- MRF-Based Deformable Registration and Ventilation Estimation of Lung CTIEEE Transactions on Medical Imaging, 2013
- Pancreatic Volume and Endocrine and Exocrine Functions in Patients With DiabetesPancreas, 2011
- Pancreatic volume in type 1 und type 2 diabetes mellitusActa Diabetologica, 2001
- Measuring agreement in method comparison studiesStatistical Methods in Medical Research, 1999
- STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENTThe Lancet, 1986