Interdisciplinary Development and Fine-Tuning of CARDIO, a Large Language Model for Cardiovascular Health Education in HIV Care: Tutorial (Preprint)

Authors:
Preprint
Publisher website
Google Scholar
Abstract
BACKGROUND The integration of artificial intelligence in health care presents a significant opportunity to revolutionize patient care. In the United States, an estimated 129 million people have at least 1 chronic illness, with 42% having 2 or more. Despite being largely preventable, the prevalence of chronic illness is expected to rise and impose significant economic burdens and financial toxicity on health care consumers. OBJECTIVE We leveraged an interdisciplinary team encompassing nursing, public health, and computer science to optimize health through prevention education for cardiovascular and metabolic comorbidities in persons living with HIV. In this tutorial, we describe the iterative, data-based development and evaluation of an intersectionality-informed large language model designed to support patient teaching in this population. METHODS First, we curated data by scraping publicly available, authoritative, evidence-based sources to capture a comprehensive dataset, supplemented by publicly available HIV forum content. Second, we benchmarked candidate large language models and generated a fine-tuning dataset using GPT-4 through multiturn question and answer conversations, using standardized metrics to assess baseline model performance. Third, we iteratively refined the selected model via low-rank adaptation and reinforcement learning, integrating quantitative metrics with qualitative expert evaluations. RESULTS Pre-existing large language models (LLMs) demonstrated poor n-gram agreement, dissonance from model answers (accuracy 4.16, readability 4.63, and professionalism 4.58), and difficult readability (Kincaid 8.54 and Jargon 4.44). After prompt adjustments and fine-tuning, preliminary results demonstrate the potential of a customized Llama-based LLM to provide personalized, culturally salient patient education. CONCLUSIONS We present a data-based, step-by-step tutorial for interdisciplinary development of CARDIO, a specialized LLM, for cardiovascular health education in HIV care. Through comprehensive data curation and scraping, systematic benchmarking, and a dual-stage fine-tuning pipeline, CARDIO’s performance improved markedly (accuracy 5.0, readability 4.98, professionalism 4.98, Kincaid 7.17, and Jargon 2.92). Although patient pilot testing remains forthcoming, our results demonstrate that targeted data curation, rigorous benchmarking, and iterative fine-tuning have provided a robust evaluation of the model’s potential. By building an LLM tailored to cardiovascular health promotion and patient education, this work lays the foundation for innovative artificial intelligence–driven strategies to manage comorbid conditions in people living with HIV.
All Related Versions

This publication has 50 references indexed in Scilit: