INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis
Preprint
Abstract
Synthesizing information from multiple data sources plays a crucial role in the practice of modern medicine. Current applications of artificial intelligence in medicine often focus on single-modality data due to a lack of publicly available, multimodal medical datasets. To address this limitation, we introduce INSPECT, which contains de-identified longitudinal records from a large cohort of patients at risk for pulmonary embolism (PE), along with ground truth labels for multiple outcomes. INSPECT contains data from 19,402 patients, including CT images, radiology report impression sections, and structured electronic health record (EHR) data (i.e. demographics, diagnoses, procedures, vitals, and medications). Using INSPECT, we develop and release a benchmark for evaluating several baseline modeling approaches on a variety of important PE related tasks. We evaluate image-only, EHR-only, and multimodal fusion models. Trained models and the de-identified dataset are made available for non-commercial use under a data use agreement. To the best of our knowledge, INSPECT is the largest multimodal dataset integrating 3D medical imaging and EHR for reproducible methods evaluation and research.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- A review on multimodal medical image fusion: Compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metricsComputers in Biology and Medicine, 2022
- Deep learning for pulmonary embolism detection on computed tomography pulmonary angiogram: a systematic review and meta-analysisScientific Reports, 2021
- Multimodal medical image fusion review: Theoretical background and recent advancesSignal Processing, 2021
- Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detectionScientific Reports, 2020
- Evaluation of acute pulmonary embolism and clot burden on CTPA with deep learningEuropean Radiology, 2020
- Multi-Channel 3D Deep Feature Learning for Survival Time Prediction of Brain Tumor Patients Using Multi-Modal NeuroimagesScientific Reports, 2019
- The UK Biobank resource with deep phenotyping and genomic dataNature, 2018
- A new contrast based multimodal medical image fusion frameworkNeurocomputing, 2015
- A novel approach for multimodal medical image fusionExpert Systems with Applications, 2014
- Severity of acute pulmonary embolism: evaluation of a new spiral CT angiographic score in correlation with echocardiographic dataEuropean Radiology, 2003