MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modelling
Preprint
Abstract
Multimodal electronic health record (EHR) data provide richer, complementary insights into patient health compared to single-modality data. However, effectively integrating diverse data modalities for clinical prediction modeling remains challenging due to the substantial data requirements. We introduce a novel architecture, Mixture-of-Multimodal-Agents (MoMA), designed to leverage multiple large language model (LLM) agents for clinical prediction tasks using multimodal EHR data. MoMA employs specialized LLM agents ("specialist agents") to convert non-textual modalities, such as medical images and laboratory results, into structured textual summaries. These summaries, together with clinical notes, are combined by another LLM ("aggregator agent") to generate a unified multimodal summary, which is then used by a third LLM ("predictor agent") to produce clinical predictions. Evaluating MoMA on three prediction tasks using real-world datasets with different modality combinations and prediction settings, MoMA outperforms current state-of-the-art methods, highlighting its enhanced accuracy and flexibility across various tasks.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Medical Large Vision Language Models with Multi-image Visual Ability2025
- Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generationNature Communications, 2025
- CXR-LLaVA: a multimodal large language model for interpreting chest X-ray imagesEuropean Radiology, 2025
- Multimodal assessment improves neuroprognosis performance in clinically unresponsive critical-care patients with brain injuryNature Medicine, 2024
- Clinical natural language processing for secondary usesJournal of Biomedical Informatics, 2024
- A framework for future national pediatric pandemic respiratory disease severity triage: The HHS pediatric COVID-19 data challengeJournal of Clinical and Translational Science, 2023
- Ethics of large language models in medicine and medical researchThe Lancet Digital Health, 2023
- Development and multimodal validation of a substance misuse algorithm for referral to treatment using artificial intelligence (SMART-AI): a retrospective deep learning studyThe Lancet Digital Health, 2022
- Harnessing multimodal data integration to advance precision oncologyNature Reviews Cancer, 2021
- The Trauma Quality Improvement Program of the American College of Surgeons Committee on TraumaJournal of the American College of Surgeons, 2009