Leveraging ChatGPT and explainable AI for enhancing clinical decision support
Authors:
Open Access
- 5 November 2025
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 15 (1) , 1-14
Abstract
Large language models (LLMs) excel in many natural language processing tasks. However, their direct application to tabular, domain-specific clinical data remains challenging, as they lack innate mechanisms for reasoning over structured numerical features. This paper presents HealthAI-Prompt, a novel framework that systematically adapts LLMs for tabular clinical decision-making-specifically, predicting diabetes risk-through contextual prompts that combine detailed task descriptions with domain knowledge. Our domain knowledge integration leverages insights from high-performing machine learning models optimized via automated machine learning (AutoML) technique, together with local explanations for representative examples. These explanations, generated by multiple methods, are rigorously evaluated using fidelity, stability, and monotonicity metrics, ensuring reliability and clinical validity. The most accurate explanations are then embedded into the prompts, enabling the LLM to interpret structured features in a clinically meaningful way without fine-tuning. This approach uniquely bridges AutoML-driven predictive modeling with LLM-based reasoning over tabular inputs, improving predictive accuracy, and transparency in healthcare AI. Through comparative analysis of HealthAI-Prompt and various AutoML techniques under diverse data conditions, we offer insights into the impact of different prompt engineering strategies on model performance. As part of our evaluation, we also include Chain-of-Thought prompting as a baseline to contextualize the gains of our proposed method.Keywords
This publication has 46 references indexed in Scilit:
- OpenMedLM: prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language modelsScientific Reports, 2024
- Can large language models reason about medical questions?Patterns, 2024
- A study of generative large language model for medical research and healthcarenpj Digital Medicine, 2023
- Evaluating large language models on medical evidence summarizationnpj Digital Medicine, 2023
- Large language models in health care: Development, applications, and challengesHealth Care Science, 2023
- TPOT-NN: augmenting tree-based automated machine learning with neural network estimatorsGenetic Programming and Evolvable Machines, 2021
- GAMA: A General Automated Machine Learning Assistant2021
- Automated Machine Learning: Techniques and Frameworks2020
- ILIME: Local and Global Interpretable Model-Agnostic Explainer of Black-Box Decision2019
- Grammar-based Genetic Programming: a surveyGenetic Programming and Evolvable Machines, 2010