MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph
Preprint
Abstract
Medical deep learning models depend heavily on domain-specific knowledge to perform well on knowledge-intensive clinical tasks. Prior work has primarily leveraged unimodal knowledge graphs, such as the Unified Medical Language System (UMLS), to enhance model performance. However, integrating multimodal medical knowledge graphs remains largely underexplored, mainly due to the lack of resources linking imaging data with clinical concepts. To address this gap, we propose MEDMKG, a Medical Multimodal Knowledge Graph that unifies visual and textual medical information through a multi-stage construction pipeline. MEDMKG fuses the rich multimodal data from MIMIC-CXR with the structured clinical knowledge from UMLS, utilizing both rule-based tools and large language models for accurate concept extraction and relationship modeling. To ensure graph quality and compactness, we introduce Neighbor-aware Filtering (NaF), a novel filtering algorithm tailored for multimodal knowledge graphs. We evaluate MEDMKG across three tasks under two experimental settings, benchmarking twenty-four baseline methods and four state-of-the-art vision-language backbones on six datasets. Results show that MEDMKG not only improves performance in downstream medical tasks but also offers a strong foundation for developing adaptive and robust strategies for multimodal knowledge integration in medical artificial intelligence.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Knowledge‐driven deep learning for fast MR imaging: Undersampled MR image reconstruction from supervised to un‐supervised learningMagnetic Resonance in Medicine, 2024
- K-PathVQA: Knowledge-Aware Multimodal Representation for Pathology Visual Question AnsweringIEEE Journal of Biomedical and Health Informatics, 2023
- Building a knowledge graph to enable precision medicineScientific Data, 2023
- Medical knowledge-based network for Patient-oriented Visual Question AnsweringInformation Processing & Management, 2022
- Richpedia: A Large-Scale, Comprehensive Multi-Modal Knowledge GraphBig Data Research, 2020
- Democratizing EHR analyses with FIDDLE: a flexible data-driven preprocessing pipeline for structured clinical dataJournal of the American Medical Informatics Association, 2020
- Constructing biomedical domain-specific knowledge graph with minimum supervisionKnowledge and Information Systems, 2019
- A dataset of clinically generated visual questions and answers about radiology imagesScientific Data, 2018
- The human disease networkProceedings of the National Academy of Sciences, 2007
- The Unified Medical Language System (UMLS): integrating biomedical terminologyNucleic Acids Research, 2004