Interdisciplinary Development and Fine-Tuning of CARDIO a LLM for Cardiovascular Health Education in HIV Care: A Tutorial (Preprint)

Authors:
Open Access
Publisher website
Google Scholar
Abstract
The integration of Artificial Intelligence in healthcare presents as a significant opportunity to revolutionize patient care. In the United States, an estimated 129 million people have at least one chronic illness, with 42% having two or more. Despite being largely preventable, the prevalence of chronic illness is expected to rise and impose significant economic burdens and financial toxicity on healthcare consumers. We leveraged an interdisciplinary team encompassing nursing, public health, and computer science to optimize health through prevention education for cardiovascular and metabolic comorbidities in persons living with HIV. In this tutorial, we describe the iterative, data-based development and evaluation of an intersectionality-informed large language model designed to support patient teaching in this population. First, we curated data by scraping publicly available, authoritative, evidence-based sources to capture a comprehensive dataset, supplemented by publicly available HIV forum content. Second, we benchmarked candidate large language models and generated a fine-tuning dataset using GPT-4 through multi-turn question-answer conversations, employing standardized metrics to assess baseline model performance. Third, we iteratively refined the selected model via Low-Rank Adaptation and reinforcement learning, integrating quantitative metrics with qualitative expert evaluations. Pre-existing LLM models demonstrated poor n-gram agreement, dissonance from model answers (Accuracy 4.16, Readability 4.63, Professionalism 4.58), and difficult readability (Kincaid 8.54, Jargon 4.44). After prompt adjustments and fine-tuning, preliminary results demonstrate the potential of a customized LLaMA-based LLM to provide personalized, culturally salient patient education. We present a data-based, step-by-step tutorial for interdisciplinary development of CARDIO, a specialized LLM, for cardiovascular health education in HIV care. Through comprehensive data curation and scraping, systematic benchmarking, and a dual-stage fine-tuning pipeline, CARDIO's performance improved markedly (Accuracy 5.0, Readability 4.98, Professionalism 4.98, Kincaid 7.17, Jargon 2.92). Although patient pilot testing remains forthcoming, our results demonstrate that targeted data curation, rigorous benchmarking, and iterative fine-tuning have provided a robust evaluation of the model's potential. By building an LLM tailored to cardiovascular health promotion and patient education, this work lays the foundation for innovative AI-driven strategies to manage comorbid conditions in people living with HIV.

This publication has 60 references indexed in Scilit: