Kabilan Elangovan
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Kabilan Elangovan has a total of 68 co-authors. Here are the top co-authors by number of shared publications:
- Daniel Shu Wei Ting
Singapore Eye Research Institute, Singapore National Eye Center, Singapore, Singapore
8 shared publications - Jasmine Chiat Ling Ong
Duke-NUS AI + Medical Sciences Initiative, Duke-NUS Medical School, Singapore, Singapore
8 shared publications Daniel Shu Wei Ting
5 shared publications- Nan Liu
Centre for Biomedical Data Science, Duke-NUS Medical School, Singapore, Singapore
5 shared publications - Yuhe Ke
Division of Anesthesiology and Perioperative Medicine, Singapore General Hospital, Singapore, Singapore
5 shared publications Liyuan Jin
Affiliations Singapore Health Services, Artificial Intelligence Office, Singapore, Singapore National Eye Centre, Singapore Eye Research Institute, Singapore, MOHH Holdings (Singapore) Private Limited, Singapore
4 shared publications
Latest Publications
- Open Access
- 18 November 2026
- Published by Elsevier in The Lancet Digital Health
Bias and equity in large language model-based applications for health care: a scoping review
- 18 November 2026
- Published by Elsevier in The Lancet Digital Health
Open AccessBias and equity in large language model-based applications for health care: a scoping review
- Open Access
- 1 September 2026
- Published by JMIR Publications Inc. in JMIR Preprints
A Generative Artificial Intelligence Assistant in Medical Accreditation Examinations (GAIA-Ex) (Preprint)
BACKGROUND Large language models (LLMs) have emerging applications in healthcare, including medical education and content generation. Specialist board examination preparation remains resource-intensive and relies heavily on manual question curation. OBJECTIVE We evaluated generative artificial intelligence (AI) as an adjunct for generating specialist board-style questions and revision material while developing a novel structured evaluation framework. METHODS The six-domain Veracity, Applicability, Coherence, Accuracy, Time-appropriateness and Elaboration (VACATE) framework was developed, with lower scores indicating more favourable evaluations. Examination-style questions and answers on prostate cancer were generated using Claude 3.7, Claude 4.1 and Claude 4.5. Three consultant urologists assessed clinical validity and examination suitability, while three urology residents evaluated readability and educational utility. Model versions were compared using Friedman tests with Holm correction, followed by exploratory pairwise Wilcoxon signed-rank tests. RESULTS Both learners and examiners provided generally favourable ratings across the VACATE domains. Among learners, aggregate mean scores improved from 2.11 for Claude 3.7 to 1.78 for both Claude 4.1 and Claude 4.5. Learner-rated Veracity and Applicability improved numerically across successive versions. Among examiners, Claude 4.5 achieved the most favourable aggregate mean score (2.10), followed by Claude 3.7 (2.13) and Claude 4.1 (2.65). Examiner ratings showed a non-linear pattern, with Claude 4.1 receiving less favourable Accuracy and Applicability ratings. Claude 4.5 demonstrated the most balanced descriptive profile across both groups. No VACATE domain differed significantly among versions after Holm correction, and exploratory pairwise comparisons were similarly non-significant. CONCLUSIONS GAIA-Ex provides a feasible approach to AI-assisted specialist examination preparation, while VACATE enables multidimensional evaluation beyond factual accuracy. Although descriptive patterns differed between model versions, the small number of raters limited statistical power; therefore, non-significance should not be interpreted as equivalence. Larger-scale validation is required.
- 1 September 2026
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessA Generative Artificial Intelligence Assistant in Medical Accreditation Examinations (GAIA-Ex) (Preprint)
BACKGROUND Large language models (LLMs) have emerging applications in healthcare, including medical education and content generation. Specialist board examination preparation remains resource-intensive and relies heavily on manual question curation. OBJECTIVE We evaluated generative artificial intelligence (AI) as an adjunct for generating specialist board-style questions and revision material while developing a novel structured evaluation framework. METHODS The six-domain Veracity, Applicability, Coherence, Accuracy, Time-appropriateness and Elaboration (VACATE) framework was developed, with lower scores indicating more favourable evaluations. Examination-style questions and answers on prostate cancer were generated using Claude 3.7, Claude 4.1 and Claude 4.5. Three consultant urologists assessed clinical validity and examination suitability, while three urology residents evaluated readability and educational utility. Model versions were compared using Friedman tests with Holm correction, followed by exploratory pairwise Wilcoxon signed-rank tests. RESULTS Both learners and examiners provided generally favourable ratings across the VACATE domains. Among learners, aggregate mean scores improved from 2.11 for Claude 3.7 to 1.78 for both Claude 4.1 and Claude 4.5. Learner-rated Veracity and Applicability improved numerically across successive versions. Among examiners, Claude 4.5 achieved the most favourable aggregate mean score (2.10), followed by Claude 3.7 (2.13) and Claude 4.1 (2.65). Examiner ratings showed a non-linear pattern, with Claude 4.1 receiving less favourable Accuracy and Applicability ratings. Claude 4.5 demonstrated the most balanced descriptive profile across both groups. No VACATE domain differed significantly among versions after Holm correction, and exploratory pairwise comparisons were similarly non-significant. CONCLUSIONS GAIA-Ex provides a feasible approach to AI-assisted specialist examination preparation, while VACATE enables multidimensional evaluation beyond factual accuracy. Although descriptive patterns differed between model versions, the small number of raters limited statistical power; therefore, non-significance should not be interpreted as equivalence. Larger-scale validation is required.
- Open Access
- 29 June 2026
- Published by Elsevier in SSRN Electronic Journal
Health AI Literacy: An Emerging Social Determinant of Health in the Age of Generative AI
Generative artificial intelligence is rapidly transforming how individuals access and interpret health information. Patients have consistently accessed these artificial intelligence (AI) systems, and surveys and chat log analyses from organizations such as OpenAI and Microsoft indicate that a substantial proportion of use already involves health-related queries. More recently, product launches from major AI labs have increasingly been designed to directly target patient-facing use cases, further reinforcing this trend. Existing constructs such as health literacy and digital health literacy do not fully capture the competencies required to safely engage with AI-generated outputs. In this paper, we propose health AI literacy as a distinct construct and emerging social determinant of health, defined as the ability to critically, safely, and effectively interact with AI-enabled health technologies. We introduce a conceptual framework comprising six domains: functional AI competence, structural understanding, interpretive competence, clinical risk awareness, data and governance awareness, and social and equity awareness. We further discuss how disparities in health AI literacy may create an "AI health divide," influencing healthcare utilization, trust in digital health systems, and health outcomes at both population and global levels. Advancing measurement, education, and policy will be critical to ensure that AIenabled healthcare improves equity rather than amplifies existing disparities.
- 29 June 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessHealth AI Literacy: An Emerging Social Determinant of Health in the Age of Generative AI
Generative artificial intelligence is rapidly transforming how individuals access and interpret health information. Patients have consistently accessed these artificial intelligence (AI) systems, and surveys and chat log analyses from organizations such as OpenAI and Microsoft indicate that a substantial proportion of use already involves health-related queries. More recently, product launches from major AI labs have increasingly been designed to directly target patient-facing use cases, further reinforcing this trend. Existing constructs such as health literacy and digital health literacy do not fully capture the competencies required to safely engage with AI-generated outputs. In this paper, we propose health AI literacy as a distinct construct and emerging social determinant of health, defined as the ability to critically, safely, and effectively interact with AI-enabled health technologies. We introduce a conceptual framework comprising six domains: functional AI competence, structural understanding, interpretive competence, clinical risk awareness, data and governance awareness, and social and equity awareness. We further discuss how disparities in health AI literacy may create an "AI health divide," influencing healthcare utilization, trust in digital health systems, and health outcomes at both population and global levels. Advancing measurement, education, and policy will be critical to ensure that AIenabled healthcare improves equity rather than amplifies existing disparities.
- Open Access
- 4 September 2025
- Published by Public Library of Science (PLoS) in PLOS Digital Health
- Vol. 4  (9) , e0000961
Development and evaluation of a lightweight large language model chatbot for medication enquiry
Large Language Models (LLMs) show promise in augmenting digital health applications. However, development and scaling of large models face computational constraints, data security concerns and limitations of internet accessibility in some regions. We developed and tested Med-Pal, a medical domain-specific LLM-chatbot fine-tuned with a fine-grained, expert curated medication-enquiry dataset consisting of 1,100 question and answer pairs. We trained and validated five light-weight, open-source LLMs of smaller parameter size (7 billion or less) on a validation dataset of 231 medication-related enquiries. We introduce SCORE, an LLM-specific evaluation criteria for clinical adjudication of LLM responses, performed by a multidisciplinary expert team. The best performing lighted-weight LLM was chosen as Med-Pal for further engineering with guard-railing against adversarial prompts. Med-Pal outperformed Biomistral and Meerkat, achieving 71.9% high-quality responses in a separate testing dataset. Med-Pal’s light-weight architecture, clinical alignment and safety guardrails enable implementation under varied settings, including those with limited digital infrastructure.
- 4 September 2025
- Published by Public Library of Science (PLoS) in PLOS Digital Health
- Vol. 4  (9) , e0000961
Open AccessDevelopment and evaluation of a lightweight large language model chatbot for medication enquiry
Large Language Models (LLMs) show promise in augmenting digital health applications. However, development and scaling of large models face computational constraints, data security concerns and limitations of internet accessibility in some regions. We developed and tested Med-Pal, a medical domain-specific LLM-chatbot fine-tuned with a fine-grained, expert curated medication-enquiry dataset consisting of 1,100 question and answer pairs. We trained and validated five light-weight, open-source LLMs of smaller parameter size (7 billion or less) on a validation dataset of 231 medication-related enquiries. We introduce SCORE, an LLM-specific evaluation criteria for clinical adjudication of LLM responses, performed by a multidisciplinary expert team. The best performing lighted-weight LLM was chosen as Med-Pal for further engineering with guard-railing against adversarial prompts. Med-Pal outperformed Biomistral and Meerkat, achieving 71.9% high-quality responses in a separate testing dataset. Med-Pal’s light-weight architecture, clinical alignment and safety guardrails enable implementation under varied settings, including those with limited digital infrastructure.
- Open Access
- 3 September 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27, e67469
Large Language Models in Randomized Controlled Trials Design: Observational Study
Background: Randomized controlled trials (RCTs) face challenges such as limited generalizability, insufficient recruitment diversity, and high failure rates, often due to restrictive eligibility criteria and inefficient patient selection. Large language models (LLMs) have shown promise in various clinical tasks, but their potential role in RCT design remains underexplored. Objective: This study investigates the ability of LLMs, specifically GPT-4-Turbo-Preview, to assist in designing RCTs that enhance generalizability, recruitment diversity, and reduce failure rates, while maintaining clinical safety and ethical standards. Methods: We conducted a noninterventional, observational study analyzing 20 parallel-arm RCTs, comprising 10 completed and 10 registered studies published after January 2024 to mitigate pretraining biases. The LLM was tasked with generating RCT designs based on input criteria, including eligibility, recruitment strategies, interventions, and outcomes. The accuracy of LLM-generated designs was quantitatively assessed by 2 independent clinical experts by comparing them to clinically validated ground truth data from ClinicalTrials.gov. We have conducted statistical analysis using natural language processing–based methods, including Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE)-L, and Metric for Evaluation of Translation with Explicit ORdering (METEOR), for objective scoring on corresponding LLM outputs. Qualitative assessments were performed using Likert scale ratings (1-3) for domains such as safety, clinical accuracy, objectivity or bias, pragmatism, inclusivity, and diversity. Results: The LLM achieved an overall accuracy of 72% in replicating RCT designs. Recruitment and intervention designs demonstrated high agreement with the ground truth, achieving 88% and 93% accuracy, respectively. However, LLMs showed lower accuracy in designing eligibility criteria (55%) and outcomes measurement (53%). Natural language processing statistical analysis reported BLEU=0.04, ROUGE-L=0.20, and METEOR=0.18 on average objective scoring of LLM outputs. Qualitative evaluations showed that LLM-generated designs scored above 2 points and closely matched the original designs in scores across all domains, indicating strong clinical alignment. Specifically, both original and LLM-based designs ranked similarly high in safety, clinical accuracy, and objectivity or bias in published RCTs. Moreover, LLM-based design ranked noninferior to original designs in registered RCTs in multiple domains. In particular, LLMs enhanced diversity and pragmatism, which are key factors in improving RCT generalizability and addressing failure rates. Conclusions: LLMs, such as GPT-4-Turbo-Preview, have demonstrated potential in improving RCT design, particularly in recruitment and intervention planning, while enhancing generalizability and addressing diversity. However, expert oversight and regulatory measures are essential to ensure patient safety and ethical standards. The findings support further integration of LLMs into clinical trial design, although continued refinement is necessary to address limitations in eligibility and outcomes measurement.
- 3 September 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27, e67469
Open AccessLarge Language Models in Randomized Controlled Trials Design: Observational Study
Background: Randomized controlled trials (RCTs) face challenges such as limited generalizability, insufficient recruitment diversity, and high failure rates, often due to restrictive eligibility criteria and inefficient patient selection. Large language models (LLMs) have shown promise in various clinical tasks, but their potential role in RCT design remains underexplored. Objective: This study investigates the ability of LLMs, specifically GPT-4-Turbo-Preview, to assist in designing RCTs that enhance generalizability, recruitment diversity, and reduce failure rates, while maintaining clinical safety and ethical standards. Methods: We conducted a noninterventional, observational study analyzing 20 parallel-arm RCTs, comprising 10 completed and 10 registered studies published after January 2024 to mitigate pretraining biases. The LLM was tasked with generating RCT designs based on input criteria, including eligibility, recruitment strategies, interventions, and outcomes. The accuracy of LLM-generated designs was quantitatively assessed by 2 independent clinical experts by comparing them to clinically validated ground truth data from ClinicalTrials.gov. We have conducted statistical analysis using natural language processing–based methods, including Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE)-L, and Metric for Evaluation of Translation with Explicit ORdering (METEOR), for objective scoring on corresponding LLM outputs. Qualitative assessments were performed using Likert scale ratings (1-3) for domains such as safety, clinical accuracy, objectivity or bias, pragmatism, inclusivity, and diversity. Results: The LLM achieved an overall accuracy of 72% in replicating RCT designs. Recruitment and intervention designs demonstrated high agreement with the ground truth, achieving 88% and 93% accuracy, respectively. However, LLMs showed lower accuracy in designing eligibility criteria (55%) and outcomes measurement (53%). Natural language processing statistical analysis reported BLEU=0.04, ROUGE-L=0.20, and METEOR=0.18 on average objective scoring of LLM outputs. Qualitative evaluations showed that LLM-generated designs scored above 2 points and closely matched the original designs in scores across all domains, indicating strong clinical alignment. Specifically, both original and LLM-based designs ranked similarly high in safety, clinical accuracy, and objectivity or bias in published RCTs. Moreover, LLM-based design ranked noninferior to original designs in registered RCTs in multiple domains. In particular, LLMs enhanced diversity and pragmatism, which are key factors in improving RCT generalizability and addressing failure rates. Conclusions: LLMs, such as GPT-4-Turbo-Preview, have demonstrated potential in improving RCT design, particularly in recruitment and intervention planning, while enhancing generalizability and addressing diversity. However, expert oversight and regulatory measures are essential to ensure patient safety and ethical standards. The findings support further integration of LLMs into clinical trial design, although continued refinement is necessary to address limitations in eligibility and outcomes measurement.
- 7 July 2025
- Published by Elsevier in British Journal of Anaesthesia
- Vol. 135  (3) , 800-802
Differential reasoning and chain-of-thought processes in Deepseek-R1 and Open AI o3-mini-high for determining American Society of Anesthesiologists physical status
- 7 July 2025
- Published by Elsevier in British Journal of Anaesthesia
- Vol. 135  (3) , 800-802
Differential reasoning and chain-of-thought processes in Deepseek-R1 and Open AI o3-mini-high for determining American Society of Anesthesiologists physical status
- Top Cited PapersOpen Access
- 28 March 2025
- Published by Springer Nature in npj Digital Medicine
- Vol. 8  (1) , 1-8
A scoping review on generative AI and large language models in mitigating medication related harm
Medication-related harm has a significant impact on global healthcare costs and patient outcomes. Generative artificial intelligence (GenAI) and large language models (LLM) have emerged as a promising tool in mitigating risks of medication-related harm. This review evaluates the scope and effectiveness of GenAI and LLM in reducing medication-related harm. We screened 4 databases for literature published from 1st January 2012 to 15th October 2024. A total of 3988 articles were identified, and 30 met the criteria for inclusion into the final review. Generative AI and LLMs were applied in three key applications: drug-drug interaction identification and prediction, clinical decision support, and pharmacovigilance. While the performance and utility of these models varied, they generally showed promise in early identification, classification of adverse drug events, and supporting decision-making for medication management. However, no studies tested these models prospectively, suggesting a need for further investigation into integration and real-world application.
- 28 March 2025
- Published by Springer Nature in npj Digital Medicine
- Vol. 8  (1) , 1-8
Top Cited PapersOpen AccessA scoping review on generative AI and large language models in mitigating medication related harm
Medication-related harm has a significant impact on global healthcare costs and patient outcomes. Generative artificial intelligence (GenAI) and large language models (LLM) have emerged as a promising tool in mitigating risks of medication-related harm. This review evaluates the scope and effectiveness of GenAI and LLM in reducing medication-related harm. We screened 4 databases for literature published from 1st January 2012 to 15th October 2024. A total of 3988 articles were identified, and 30 met the criteria for inclusion into the final review. Generative AI and LLMs were applied in three key applications: drug-drug interaction identification and prediction, clinical decision support, and pharmacovigilance. While the performance and utility of these models varied, they generally showed promise in early identification, classification of adverse drug events, and supporting decision-making for medication management. However, no studies tested these models prospectively, suggesting a need for further investigation into integration and real-world application.
- Open Access
- 12 October 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Large Language Models in Randomized Controlled Trials Design (Preprint)
BACKGROUND Randomized controlled trials (RCTs) face challenges such as limited generalizability, insufficient recruitment diversity, and high failure rates, often due to restrictive eligibility criteria and inefficient patient selection. Large language models (LLMs) have shown promise in various clinical tasks, but their potential role in RCT design remains underexplored. OBJECTIVE This study investigates the ability of LLMs, specifically GPT-4-Turbo-Preview, to assist in designing RCTs that enhance generalizability, recruitment diversity, and reduce failure rates, while maintaining clinical safety and ethical standards. METHODS We conducted a non-interventional, observational study analyzing 20 parallel-arm RCTs, comprising 10 completed and 10 ongoing studies published after January 2024 to mitigate pretraining biases. The LLM was tasked with generating RCT designs based on input criteria, including eligibility, recruitment strategies, interventions, and outcomes. The accuracy of LLM-generated designs was quantitatively assessed by comparing them to clinically validated ground truth data from ClinicalTrials.gov. Qualitative assessments were performed using Likert scale ratings (1–3) for domains such as safety, accuracy, objectivity, pragmatism, inclusivity, and diversity. RESULTS The LLM achieved an overall accuracy of 72% in replicating RCT designs. Recruitment and intervention designs demonstrated high agreement with the ground truth, achieving 88% and 93% accuracy, respectively. However, LLMs showed lower accuracy in designing eligibility criteria (55%) and outcomes measurement (53%). Qualitative evaluations showed that LLM-generated designs scored above 2 points across all domains, indicating strong clinical alignment. In particular, LLMs enhanced diversity and pragmatism, which are key factors in improving RCT generalizability and addressing failure rates. CONCLUSIONS LLMs, such as GPT-4-Turbo-Preview, have demonstrated potential in improving RCT design, particularly in recruitment and intervention planning, while enhancing generalizability and addressing diversity. However, expert oversight and regulatory measures are essential to ensure patient safety and ethical standards. The findings support further integration of LLMs into clinical trial design, although continued refinement is necessary to address limitations in eligibility and outcomes measurement.
- 12 October 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessLarge Language Models in Randomized Controlled Trials Design (Preprint)
BACKGROUND Randomized controlled trials (RCTs) face challenges such as limited generalizability, insufficient recruitment diversity, and high failure rates, often due to restrictive eligibility criteria and inefficient patient selection. Large language models (LLMs) have shown promise in various clinical tasks, but their potential role in RCT design remains underexplored. OBJECTIVE This study investigates the ability of LLMs, specifically GPT-4-Turbo-Preview, to assist in designing RCTs that enhance generalizability, recruitment diversity, and reduce failure rates, while maintaining clinical safety and ethical standards. METHODS We conducted a non-interventional, observational study analyzing 20 parallel-arm RCTs, comprising 10 completed and 10 ongoing studies published after January 2024 to mitigate pretraining biases. The LLM was tasked with generating RCT designs based on input criteria, including eligibility, recruitment strategies, interventions, and outcomes. The accuracy of LLM-generated designs was quantitatively assessed by comparing them to clinically validated ground truth data from ClinicalTrials.gov. Qualitative assessments were performed using Likert scale ratings (1–3) for domains such as safety, accuracy, objectivity, pragmatism, inclusivity, and diversity. RESULTS The LLM achieved an overall accuracy of 72% in replicating RCT designs. Recruitment and intervention designs demonstrated high agreement with the ground truth, achieving 88% and 93% accuracy, respectively. However, LLMs showed lower accuracy in designing eligibility criteria (55%) and outcomes measurement (53%). Qualitative evaluations showed that LLM-generated designs scored above 2 points across all domains, indicating strong clinical alignment. In particular, LLMs enhanced diversity and pragmatism, which are key factors in improving RCT generalizability and addressing failure rates. CONCLUSIONS LLMs, such as GPT-4-Turbo-Preview, have demonstrated potential in improving RCT design, particularly in recruitment and intervention planning, while enhancing generalizability and addressing diversity. However, expert oversight and regulatory measures are essential to ensure patient safety and ethical standards. The findings support further integration of LLMs into clinical trial design, although continued refinement is necessary to address limitations in eligibility and outcomes measurement.
- Open Access
- 24 July 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26, e57721
Comparison of the Quality of Discharge Letters Written by Large Language Models and Junior Clinicians: Single-Blinded Study
Background: Discharge letters are a critical component in the continuity of care between specialists and primary care providers. However, these letters are time-consuming to write, underprioritized in comparison to direct clinical care, and are often tasked to junior doctors. Prior studies assessing the quality of discharge summaries written for inpatient hospital admissions show inadequacies in many domains. Large language models such as GPT have the ability to summarize large volumes of unstructured free text such as electronic medical records and have the potential to automate such tasks, providing time savings and consistency in quality. Objective: The aim of this study was to assess the performance of GPT-4 in generating discharge letters written from urology specialist outpatient clinics to primary care providers and to compare their quality against letters written by junior clinicians. Methods: Fictional electronic records were written by physicians simulating 5 common urology outpatient cases with long-term follow-up. Records comprised simulated consultation notes, referral letters and replies, and relevant discharge summaries from inpatient admissions. GPT-4 was tasked to write discharge letters for these cases with a specified target audience of primary care providers who would be continuing the patient’s care. Prompts were written for safety, content, and style. Concurrently, junior clinicians were provided with the same case records and instructional prompts. GPT-4 output was assessed for instances of hallucination. A blinded panel of primary care physicians then evaluated the letters using a standardized questionnaire tool. Results: GPT-4 outperformed human counterparts in information provision (mean 4.32, SD 0.95 vs 3.70, SD 1.27; P=.03) and had no instances of hallucination. There were no statistically significant differences in the mean clarity (4.16, SD 0.95 vs 3.68, SD 1.24; P=.12), collegiality (4.36, SD 1.00 vs 3.84, SD 1.22; P=.05), conciseness (3.60, SD 1.12 vs 3.64, SD 1.27; P=.71), follow-up recommendations (4.16, SD 1.03 vs 3.72, SD 1.13; P=.08), and overall satisfaction (3.96, SD 1.14 vs 3.62, SD 1.34; P=.36) between the letters generated by GPT-4 and humans, respectively. Conclusions: Discharge letters written by GPT-4 had equivalent quality to those written by junior clinicians, without any hallucinations. This study provides a proof of concept that large language models can be useful and safe tools in clinical documentation.
- 24 July 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26, e57721
Open AccessComparison of the Quality of Discharge Letters Written by Large Language Models and Junior Clinicians: Single-Blinded Study
Background: Discharge letters are a critical component in the continuity of care between specialists and primary care providers. However, these letters are time-consuming to write, underprioritized in comparison to direct clinical care, and are often tasked to junior doctors. Prior studies assessing the quality of discharge summaries written for inpatient hospital admissions show inadequacies in many domains. Large language models such as GPT have the ability to summarize large volumes of unstructured free text such as electronic medical records and have the potential to automate such tasks, providing time savings and consistency in quality. Objective: The aim of this study was to assess the performance of GPT-4 in generating discharge letters written from urology specialist outpatient clinics to primary care providers and to compare their quality against letters written by junior clinicians. Methods: Fictional electronic records were written by physicians simulating 5 common urology outpatient cases with long-term follow-up. Records comprised simulated consultation notes, referral letters and replies, and relevant discharge summaries from inpatient admissions. GPT-4 was tasked to write discharge letters for these cases with a specified target audience of primary care providers who would be continuing the patient’s care. Prompts were written for safety, content, and style. Concurrently, junior clinicians were provided with the same case records and instructional prompts. GPT-4 output was assessed for instances of hallucination. A blinded panel of primary care physicians then evaluated the letters using a standardized questionnaire tool. Results: GPT-4 outperformed human counterparts in information provision (mean 4.32, SD 0.95 vs 3.70, SD 1.27; P=.03) and had no instances of hallucination. There were no statistically significant differences in the mean clarity (4.16, SD 0.95 vs 3.68, SD 1.24; P=.12), collegiality (4.36, SD 1.00 vs 3.84, SD 1.22; P=.05), conciseness (3.60, SD 1.12 vs 3.64, SD 1.27; P=.71), follow-up recommendations (4.16, SD 1.03 vs 3.72, SD 1.13; P=.08), and overall satisfaction (3.96, SD 1.14 vs 3.62, SD 1.34; P=.36) between the letters generated by GPT-4 and humans, respectively. Conclusions: Discharge letters written by GPT-4 had equivalent quality to those written by junior clinicians, without any hallucinations. This study provides a proof of concept that large language models can be useful and safe tools in clinical documentation.
- Open Access
- 25 February 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Comparison of the Quality of Discharge Letters Written by Large Language Models and Junior Clinicians: Single-Blinded Study (Preprint)
BACKGROUND Discharge letters are a critical component in the continuity of care between specialists and primary care providers. However, these letters are time-consuming to write, underprioritized in comparison to direct clinical care, and are often tasked to junior doctors. Prior studies assessing the quality of discharge summaries written for inpatient hospital admissions show inadequacies in many domains. Large language models such as GPT have the ability to summarize large volumes of unstructured free text such as electronic medical records and have the potential to automate such tasks, providing time savings and consistency in quality. OBJECTIVE The aim of this study was to assess the performance of GPT-4 in generating discharge letters written from urology specialist outpatient clinics to primary care providers and to compare their quality against letters written by junior clinicians. METHODS Fictional electronic records were written by physicians simulating 5 common urology outpatient cases with long-term follow-up. Records comprised simulated consultation notes, referral letters and replies, and relevant discharge summaries from inpatient admissions. GPT-4 was tasked to write discharge letters for these cases with a specified target audience of primary care providers who would be continuing the patient’s care. Prompts were written for safety, content, and style. Concurrently, junior clinicians were provided with the same case records and instructional prompts. GPT-4 output was assessed for instances of hallucination. A blinded panel of primary care physicians then evaluated the letters using a standardized questionnaire tool. RESULTS GPT-4 outperformed human counterparts in information provision (mean 4.32, SD 0.95 vs 3.70, SD 1.27; P=.03) and had no instances of hallucination. There were no statistically significant differences in the mean clarity (4.16, SD 0.95 vs 3.68, SD 1.24; P=.12), collegiality (4.36, SD 1.00 vs 3.84, SD 1.22; P=.05), conciseness (3.60, SD 1.12 vs 3.64, SD 1.27; P=.71), follow-up recommendations (4.16, SD 1.03 vs 3.72, SD 1.13; P=.08), and overall satisfaction (3.96, SD 1.14 vs 3.62, SD 1.34; P=.36) between the letters generated by GPT-4 and humans, respectively. CONCLUSIONS Discharge letters written by GPT-4 had equivalent quality to those written by junior clinicians, without any hallucinations. This study provides a proof of concept that large language models can be useful and safe tools in clinical documentation.
- 25 February 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessComparison of the Quality of Discharge Letters Written by Large Language Models and Junior Clinicians: Single-Blinded Study (Preprint)
BACKGROUND Discharge letters are a critical component in the continuity of care between specialists and primary care providers. However, these letters are time-consuming to write, underprioritized in comparison to direct clinical care, and are often tasked to junior doctors. Prior studies assessing the quality of discharge summaries written for inpatient hospital admissions show inadequacies in many domains. Large language models such as GPT have the ability to summarize large volumes of unstructured free text such as electronic medical records and have the potential to automate such tasks, providing time savings and consistency in quality. OBJECTIVE The aim of this study was to assess the performance of GPT-4 in generating discharge letters written from urology specialist outpatient clinics to primary care providers and to compare their quality against letters written by junior clinicians. METHODS Fictional electronic records were written by physicians simulating 5 common urology outpatient cases with long-term follow-up. Records comprised simulated consultation notes, referral letters and replies, and relevant discharge summaries from inpatient admissions. GPT-4 was tasked to write discharge letters for these cases with a specified target audience of primary care providers who would be continuing the patient’s care. Prompts were written for safety, content, and style. Concurrently, junior clinicians were provided with the same case records and instructional prompts. GPT-4 output was assessed for instances of hallucination. A blinded panel of primary care physicians then evaluated the letters using a standardized questionnaire tool. RESULTS GPT-4 outperformed human counterparts in information provision (mean 4.32, SD 0.95 vs 3.70, SD 1.27; P=.03) and had no instances of hallucination. There were no statistically significant differences in the mean clarity (4.16, SD 0.95 vs 3.68, SD 1.24; P=.12), collegiality (4.36, SD 1.00 vs 3.84, SD 1.22; P=.05), conciseness (3.60, SD 1.12 vs 3.64, SD 1.27; P=.71), follow-up recommendations (4.16, SD 1.03 vs 3.72, SD 1.13; P=.08), and overall satisfaction (3.96, SD 1.14 vs 3.62, SD 1.34; P=.36) between the letters generated by GPT-4 and humans, respectively. CONCLUSIONS Discharge letters written by GPT-4 had equivalent quality to those written by junior clinicians, without any hallucinations. This study provides a proof of concept that large language models can be useful and safe tools in clinical documentation.