A clinically accessible small multimodal radiology model and evaluation metric for chest X-ray findings
Top Cited Papers
Open Access
- 1 April 2025
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-15
Abstract
Large foundation models show promise in biomedicine but face challenges in clinical use due to performance gaps, accessibility, cost, and lack of scalable evaluation. Here we show that open-source small multimodal models can bridge these gaps in radiology by generating free-text findings from chest X-ray images. Our data-centric approach leverages 697K curated radiology image-text pairs to train a specialized, domain-adapted chest X-ray encoder. We integrate this encoder with pre-trained language models via a lightweight adapter that aligns image and text modalities. To enable robust, clinically relevant evaluation, we develop and validate CheXprompt, a GPT-4-based metric for assessing factual accuracy aligned with radiologists’ evaluations. Benchmarked with CheXprompt and other standard factuality metrics, LLaVA-Rad (7B) achieves state-of-the-art performance, outperforming much larger models like GPT-4V and Med-PaLM M (84B). While not immediately ready for real-time clinical deployment, LLaVA-Rad is a scalable, privacy-preserving and cost-effective step towards clinically adaptable multimodal AI for radiology.Keywords
All Related Versions
Funding Information
- Microsoft Research
This publication has 44 references indexed in Scilit:
- A whole-slide foundation model for digital pathology from real-world dataNature, 2024
- Generative Artificial Intelligence for Chest Radiograph Interpretation in the Emergency DepartmentJAMA Network Open, 2023
- MKCL: Medical Knowledge with Contrastive Learning model for radiology report generationJournal of Biomedical Informatics, 2023
- The ecological footprint of medical AIEuropean Radiology, 2023
- Evaluating progress in automatic chest X-ray radiology report generationPatterns, 2023
- Large language models in medicineNature Medicine, 2023
- Robust and data-efficient generalization of self-supervised machine learning for diagnostic imagingNature Biomedical Engineering, 2023
- Foundation models for generalist medical artificial intelligenceNature, 2023
- PadChest: A large chest x-ray image dataset with multi-label annotated reportsMedical Image Analysis, 2020
- Forty-One Million RADPEER Reviews Later: What We Have Learned and Are Still LearningJournal of the American College of Radiology, 2020