Multimodal Generative AI in Diagnostics: Bridging Medical Imaging and Clinical Reasoning
Preprint
- 20 August 2025
- preprint
- Published by MDPI AG in Preprints.org
Abstract
Multimodal generative artificial intelligence (AI) has emerged as a transformative approach in medical diagnostics, integrating diverse data sources to significantly enhance clinical decision-making and patient care. In this review, we systematically analyze recent advancements and methodologies in multimodal generative AI, focusing particularly on the fusion of medical imaging data with clinical records, genomic information, and textual narratives. We evaluate how these combined modalities closely mimic physician cognitive processes, leading to improved diagnostic accuracy and personalized patient management across various specialties including radiology, pathology, dermatology, and ophthalmology. Specifically, we discuss three key integration strategies: tool-use approaches, where large language models orchestrate specialized diagnostic modules; grafting techniques, which directly incorporate visual analysis into linguistic frameworks; and unified frameworks, providing simultaneous multimodal data processing within cohesive models. Additionally, we highlight exemplary models, such as PathChat, demonstrating substantial accuracy improvements (e.g., 89.5% in pathological image interpretation) resulting from multimodal integration. We also critically assess ongoing challenges, including technical barriers to data integration, interpretability issues affecting clinical trust, privacy and ethical concerns, and the evolving regulatory landscape surrounding AI-driven diagnostics. Finally, we propose directions for future research, emphasizing the need for large-scale clinical validation studies, standardized evaluation frameworks, advances in explainable AI methods, and privacy-preserving techniques such as federated learning. Ultimately, multimodal generative AI holds significant promise to augment rather than replace clinical expertise, serving as a powerful complement to human decision-making in medicine.Keywords
All Related Versions
This publication has 66 references indexed in Scilit:
- Efficiency and Quality of Generative AI–Assisted Radiograph ReportingJAMA Network Open, 2025
- Applications of artificial intelligence in abdominal imagingAbdominal Radiology, 2025
- A Current Review of Generative AI in Medicine: Core Concepts, Applications, and Current LimitationsCurrent Reviews in Musculoskeletal Medicine, 2025
- Beyond traditional orthopaedic data analysis: AI, multimodal models and continuous monitoringKnee Surgery, Sports Traumatology, Arthroscopy, 2025
- CXR-LLaVA: a multimodal large language model for interpreting chest X-ray imagesEuropean Radiology, 2025
- Assessing GPT-4 multimodal performance in radiological image analysisEuropean Radiology, 2024
- Artificial Intelligence Applied to clinical trials: opportunities and challengesHealth and Technology, 2023
- Artificial Intelligence in Breast Cancer ScreeningJAMA Internal Medicine, 2022
- The importance of multi-modal imaging and clinical information for humans and AI-based algorithms to classify breast masses (INSPiRED 003): an international, multicenter analysisEuropean Radiology, 2022
- Randomized Clinical Trials of Artificial IntelligenceJAMA, 2020