Rachel C. Forcino
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Rachel C. Forcino has a total of 192 co-authors. Here are the top co-authors by number of shared publications:
- Glyn Elwyn
Dartmouth Health, 1 Medical Center Dr, Lebanon, NH, 03766, United States
14 shared publications - Marie-Anne Durand
Department of Ambulatory Care, Unisanté University Center for Primary Care and Public Health, University of Lausanne Lausanne Vaud Switzerland
7 shared publications - Danielle Schubbe
The Dartmouth Institute for Health Policy & Clinical Practice, Dartmouth College, Lebanon, USA
5 shared publications - Mary C. Politi
Washington University
4 shared publications - Peter Scalia
The Dartmouth Institute for Health Policy and Clinical Practice, Geisel School of Medicine at Dartmouth, Williamson Translational Building, One Medical Center Drive, Lebanon, NH 03756, USA
4 shared publications - Brant J. Oliver
Office of Care Experience, the Value Institute, Dartmouth Health, Lebanon, NH, USA
3 shared publications
Latest Publications
- 23 September 2026
- Published by American Society of Clinical Oncology (ASCO) in JCO Oncology Practice
Financial Toxicity and Quality of Life After Bone Marrow Transplant: A Longitudinal, Prospective Study
PURPOSE: Financial toxicity is common after bone marrow transplant (BMT) and is associated with worse quality of life, but longitudinal patterns remain poorly understood. METHODS: We conducted a prospective cohort study of insured adults undergoing allogeneic BMT (2018-2023). Financial toxicity (Comprehensive Score for Financial Toxicity-Functional Assessment of Chronic Illness Therapy) and quality of life (Functional Assessment of Cancer Therapy [FACT]–General, FACT-BMT) were assessed at baseline and at days (D) 90, 180, and 365. Longitudinal trends were analyzed using mixed-effects models. Relative risks (RR) for financial toxicity and hazard ratios (HR) for post-BMT mortality were estimated using generalized linear mixed-effects and time-varying Cox models. RESULTS: Among 214 transplanted patients, the median age was 58 years (IQR 46-66 year), 62% were male, and 80% were White; 47% had leukemia, 27% myelodysplastic syndrome, and 9% lymphoma; and 31% died by D365. At baseline, 56% reported none/minimal, 26% mild, and 18% moderate/high financial toxicity. Financial toxicity was associated with more cost-cutting behaviors, including reduced spending on food or clothing, use of savings, and prescription nonadherence. Expected probability of ≥mild financial toxicity reduced over time (39% baseline, 34% D90, 30% D180, 26% D365). Across time points, age >65 years and income >$90,000 in US dollars (USD) had 66% and 67% decreased risk of financial toxicity, respectively, compared with counterparts (n = 148, RR, 0.34, 95% CI [0.19 to 0.59]; RR, 0.33 [0.20 to 0.57]). Quality of life (n = 119 alive and with data by D365) with none/minimal or mild financial toxicity improved after transplant compared with baseline but significantly worsened for those with moderate/high financial toxicity over 1 year. Following adjustment for covariables, financial toxicity was not significantly associated with premature death after transplant (n = 214, HR, 1.10 [0.62-1.96]). CONCLUSION: Financial toxicity improved overtime with BMT, but persistent financial hardship was associated with quality of life. These findings support longitudinal screening and financial interventions across the transplant continuum.
- 23 September 2026
- Published by American Society of Clinical Oncology (ASCO) in JCO Oncology Practice
Financial Toxicity and Quality of Life After Bone Marrow Transplant: A Longitudinal, Prospective Study
PURPOSE: Financial toxicity is common after bone marrow transplant (BMT) and is associated with worse quality of life, but longitudinal patterns remain poorly understood. METHODS: We conducted a prospective cohort study of insured adults undergoing allogeneic BMT (2018-2023). Financial toxicity (Comprehensive Score for Financial Toxicity-Functional Assessment of Chronic Illness Therapy) and quality of life (Functional Assessment of Cancer Therapy [FACT]–General, FACT-BMT) were assessed at baseline and at days (D) 90, 180, and 365. Longitudinal trends were analyzed using mixed-effects models. Relative risks (RR) for financial toxicity and hazard ratios (HR) for post-BMT mortality were estimated using generalized linear mixed-effects and time-varying Cox models. RESULTS: Among 214 transplanted patients, the median age was 58 years (IQR 46-66 year), 62% were male, and 80% were White; 47% had leukemia, 27% myelodysplastic syndrome, and 9% lymphoma; and 31% died by D365. At baseline, 56% reported none/minimal, 26% mild, and 18% moderate/high financial toxicity. Financial toxicity was associated with more cost-cutting behaviors, including reduced spending on food or clothing, use of savings, and prescription nonadherence. Expected probability of ≥mild financial toxicity reduced over time (39% baseline, 34% D90, 30% D180, 26% D365). Across time points, age >65 years and income >$90,000 in US dollars (USD) had 66% and 67% decreased risk of financial toxicity, respectively, compared with counterparts (n = 148, RR, 0.34, 95% CI [0.19 to 0.59]; RR, 0.33 [0.20 to 0.57]). Quality of life (n = 119 alive and with data by D365) with none/minimal or mild financial toxicity improved after transplant compared with baseline but significantly worsened for those with moderate/high financial toxicity over 1 year. Following adjustment for covariables, financial toxicity was not significantly associated with premature death after transplant (n = 214, HR, 1.10 [0.62-1.96]). CONCLUSION: Financial toxicity improved overtime with BMT, but persistent financial hardship was associated with quality of life. These findings support longitudinal screening and financial interventions across the transplant continuum.
- Open Access
- 27 May 2026
- Published by Oxford University Press (OUP) in The Oncologist
- Vol. 31 (7)
Advanced cancer treatment decisions and their patient-facing financial burdens: a thematic analysis
Background: Most research about the high costs of cancer care has focused on characterizing and mitigating the financial hardship patients and families experience after cancer-directed therapy. Few studies have examined how these costs prospectively influence treatment decision-making. Even fewer address how costs impact decisions about symptom-directed therapies. We aim to describe how patient-facing costs of care influence decisions about cancer-directed and symptom-directed therapies for advanced cancer. Materials and methods: We analyzed semi-structured interviews with patients, care partners, clinicians, and health system leaders and clinical observations collected by investigators at 3 National Cancer Institute designated US cancer centers between July 2019 and May 2021. Two analysts conducted thematic analysis based on the Model of Financial Burden After Cancer Diagnosis. Results: Analysis of 157 interviews and 92 observations identified 4 themes: (1) insurance influenced treatment choice and location; (2) medical costs influenced treatment choice, location, and timing; (3) non-medical costs influenced treatment location and timing; and (4) employment changes influenced treatment choice and timing. Patients and care partners emphasized costs of symptom-directed therapies; staff focused on costs of cancer-directed therapies. Patients and care partners described how employment influenced decisions about starting or continuing cancer-directed therapy, while staff noted how employment affected decisions about symptom-directed therapy. Conclusions: Patient-facing costs profoundly influence treatment decisions in advanced cancer care, but differences in perspective among patients, care partners, and clinicians can lead to oversights in treatment decision-making. Eliciting and incorporating patients’ varied priorities into treatment decisions can minimize these oversights, better align treatment with goals of care, and reduce financial hardship.
- 27 May 2026
- Published by Oxford University Press (OUP) in The Oncologist
- Vol. 31 (7)
Open AccessAdvanced cancer treatment decisions and their patient-facing financial burdens: a thematic analysis
Background: Most research about the high costs of cancer care has focused on characterizing and mitigating the financial hardship patients and families experience after cancer-directed therapy. Few studies have examined how these costs prospectively influence treatment decision-making. Even fewer address how costs impact decisions about symptom-directed therapies. We aim to describe how patient-facing costs of care influence decisions about cancer-directed and symptom-directed therapies for advanced cancer. Materials and methods: We analyzed semi-structured interviews with patients, care partners, clinicians, and health system leaders and clinical observations collected by investigators at 3 National Cancer Institute designated US cancer centers between July 2019 and May 2021. Two analysts conducted thematic analysis based on the Model of Financial Burden After Cancer Diagnosis. Results: Analysis of 157 interviews and 92 observations identified 4 themes: (1) insurance influenced treatment choice and location; (2) medical costs influenced treatment choice, location, and timing; (3) non-medical costs influenced treatment location and timing; and (4) employment changes influenced treatment choice and timing. Patients and care partners emphasized costs of symptom-directed therapies; staff focused on costs of cancer-directed therapies. Patients and care partners described how employment influenced decisions about starting or continuing cancer-directed therapy, while staff noted how employment affected decisions about symptom-directed therapy. Conclusions: Patient-facing costs profoundly influence treatment decisions in advanced cancer care, but differences in perspective among patients, care partners, and clinicians can lead to oversights in treatment decision-making. Eliciting and incorporating patients’ varied priorities into treatment decisions can minimize these oversights, better align treatment with goals of care, and reduce financial hardship.
- Open Access
- 22 September 2025
- Published by Elsevier in Patient Education and Counseling
- Vol. 142, 109362
Automating the Observer OPTION-5 measure of shared decision making: Assessing validity by comparing large language models to human ratings
- 22 September 2025
- Published by Elsevier in Patient Education and Counseling
- Vol. 142, 109362
Open AccessAutomating the Observer OPTION-5 measure of shared decision making: Assessing validity by comparing large language models to human ratings
- Open Access
- 12 August 2025
- Published by Elsevier in International Journal of Radiation Oncology - Biology - Physics (IJROBP)
- Vol. 124 (1) , 29-35
TEMPORARY REMOVAL: The Significance of the P Value
- 12 August 2025
- Published by Elsevier in International Journal of Radiation Oncology - Biology - Physics (IJROBP)
- Vol. 124 (1) , 29-35
Open AccessTEMPORARY REMOVAL: The Significance of the P Value
- Open Access
- 13 June 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e72276
Telehealth Modality Preferences for Video and Voice-Only Visits Among US Clinicians and the Public: Cross-Sectional, Web-Based Survey Study
Background: The onset of the COVID-19 pandemic led US health systems to rapidly implement telehealth services to connect patients and clinicians. This rapid expansion of telehealth allowed us to explore how a telehealth experience may be best delivered across populations and contexts. Objective: We aimed to identify telehealth modality preferences (voice vs video) among clinicians and the populations they serve and explore the barriers to telehealth use from clinician and public perspectives. Methods: We conducted 2 independent, cross-sectional, web-based surveys. US physicians specializing in internal medicine, psychiatry, urology, orthopedic surgery, or obstetrics and gynecology completed a 23-item survey through Sermo’s research panel. Quotas ensured equal representation across the selected clinical specialties. Adult members of the US general public completed a 26-item survey through Qualtrics’ research panel. Quotas ensured the general public sample approximated the US population on educational attainment, gender, and rural residence. Results: We recruited 253 clinician participants and 418 general public participants in September 2020, with survey completion rates of 22% (253/1139) and 93% (418/451), respectively. For the initial encounter, 85% (216/253; 95% CI 80%‐89%) of clinicians and 51% (215/418; 95% CI 47%‐56%) of public participants preferred video over voice only. In multiple logistic regression analyses, members of the public with broadband internet at home were more likely than those without broadband to prefer video over voice only for a first visit with a new clinician (57% vs 40%; odds ratio 2.09, 95% CI 1.25‐3.49). In an established clinical relationship, 63% (160/253; 95% CI 57%‐69%) of clinicians and 33% (137/418; 95% CI 28%‐38%) of the general public preferred video over voice only when discussing a new clinical problem. For a follow-up visit, only 26% (65/253; 95% CI 20%‐32%) of clinicians and 28% (117/418; 95% CI 24%‐33%) of the general public preferred video over voice only. Clinicians and the general public agree that technology not working properly is their main source of telehealth frustration (86/249, 35% of clinicians; 62/220, 28% of public with telehealth experience). Other major frustrations include limitations on what content can be included in the visit (main frustration for 83/249, 33% of clinicians; 54/220, 25% of the public) and downloading new technology (52/220, 24% of the public). Conclusions: Although telehealth connections with video are increasingly common, important factors are associated with a preference for voice-only telehealth connections. Clinicians prefer video over voice-only connections more than patients do for new clinical relationships and new clinical problems. For follow-up care, both clinicians and the public prefer voice-only telephone visits over video. Barriers to video-enabled telehealth experiences include technology failures, varied technology platforms across providers, and a need for more reliable high-speed internet connection. Voice-only telephone-mediated services can potentially improve health care access and experiences in light of these barriers to video-based care.
- 13 June 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e72276
Open AccessTelehealth Modality Preferences for Video and Voice-Only Visits Among US Clinicians and the Public: Cross-Sectional, Web-Based Survey Study
Background: The onset of the COVID-19 pandemic led US health systems to rapidly implement telehealth services to connect patients and clinicians. This rapid expansion of telehealth allowed us to explore how a telehealth experience may be best delivered across populations and contexts. Objective: We aimed to identify telehealth modality preferences (voice vs video) among clinicians and the populations they serve and explore the barriers to telehealth use from clinician and public perspectives. Methods: We conducted 2 independent, cross-sectional, web-based surveys. US physicians specializing in internal medicine, psychiatry, urology, orthopedic surgery, or obstetrics and gynecology completed a 23-item survey through Sermo’s research panel. Quotas ensured equal representation across the selected clinical specialties. Adult members of the US general public completed a 26-item survey through Qualtrics’ research panel. Quotas ensured the general public sample approximated the US population on educational attainment, gender, and rural residence. Results: We recruited 253 clinician participants and 418 general public participants in September 2020, with survey completion rates of 22% (253/1139) and 93% (418/451), respectively. For the initial encounter, 85% (216/253; 95% CI 80%‐89%) of clinicians and 51% (215/418; 95% CI 47%‐56%) of public participants preferred video over voice only. In multiple logistic regression analyses, members of the public with broadband internet at home were more likely than those without broadband to prefer video over voice only for a first visit with a new clinician (57% vs 40%; odds ratio 2.09, 95% CI 1.25‐3.49). In an established clinical relationship, 63% (160/253; 95% CI 57%‐69%) of clinicians and 33% (137/418; 95% CI 28%‐38%) of the general public preferred video over voice only when discussing a new clinical problem. For a follow-up visit, only 26% (65/253; 95% CI 20%‐32%) of clinicians and 28% (117/418; 95% CI 24%‐33%) of the general public preferred video over voice only. Clinicians and the general public agree that technology not working properly is their main source of telehealth frustration (86/249, 35% of clinicians; 62/220, 28% of public with telehealth experience). Other major frustrations include limitations on what content can be included in the visit (main frustration for 83/249, 33% of clinicians; 54/220, 25% of the public) and downloading new technology (52/220, 24% of the public). Conclusions: Although telehealth connections with video are increasingly common, important factors are associated with a preference for voice-only telehealth connections. Clinicians prefer video over voice-only connections more than patients do for new clinical relationships and new clinical problems. For follow-up care, both clinicians and the public prefer voice-only telephone visits over video. Barriers to video-enabled telehealth experiences include technology failures, varied technology platforms across providers, and a need for more reliable high-speed internet connection. Voice-only telephone-mediated services can potentially improve health care access and experiences in light of these barriers to video-based care.
- Open Access
- 2 June 2025
- Published by Elsevier in PEC Innovation
- Vol. 6, 100407
A novel outreach approach for identification of familial hypercholesterolemia: Interview-based formative evaluation to improve healthcare access and quality
- 2 June 2025
- Published by Elsevier in PEC Innovation
- Vol. 6, 100407
Open AccessA novel outreach approach for identification of familial hypercholesterolemia: Interview-based formative evaluation to improve healthcare access and quality
- Open Access
- 5 November 2024
- Published by Springer Nature in Implementation Science
- Vol. 19 (1) , 1-13
The UPFRONT project: tailored implementation and evaluation of a patient decision aid to support shared decision-making about management of symptomatic uterine fibroids
To evaluate implementation of a patient decision aid for symptomatic uterine fibroid management to improve shared decision-making at five clinical settings across the United States. We used a type 3 hybrid effectiveness-implementation stepped-wedge design and the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) planning and evaluation framework. We conducted clinician training, monthly reach tracking with feedback to site clinical leads, patient and clinician surveys, and visit audio-recordings. Implementation strategies included assessment of organizational readiness for shared decision-making, synchronous clinician training, audit and feedback of decision aid reach, and access to multiple decision aid formats. Outcomes and analyses included patient-level reach, clinician-level adoption, and associations of patient-reported decision aid exposure (as treated) and setting-level implementation (intention-to-treat) with patient-reported (collaboRATE measure) and observed (OPTION-5 measure) shared decision-making. We also designed and assessed setting-level plans for sustainability and other factors impacting sustained decision aid use. The decision aid was adopted by 72 of the 74 eligible gynecologists (97%) and reached 2553 patients across five settings. CollaboRATE scores improved among patients who reported receiving the decision aid (as-treated analysis, 69% vs. 59%; OR 1.6, 95% CI 1.16–2.27). CollaboRATE scores remained consistent before and after setting-level decision aid implementation (intention-to-treat analysis, 64% vs. 63%; OR 0.86, 95% CI 0.61–1.22). Participants would prefer to receive a decision aid at multiple time points (91.9% before the visit, 90.7% during the visit, 86.5% after the visit). Shared decision-making experiences did not improve when comparing pre vs. post-implementation collaboRATE scores across included settings (intention-to-treat, 64% vs. 63%; OR 0.86, 95% CI 0.61–1.22). When patients with symptomatic uterine fibroids are given decision aids, they report higher shared decision-making scores. However, the differences we observed between the as-treated and intention-to-treat results suggest that unaddressed implementation challenges continue to limit the extent to which patients receive decision aids and likely hinder their overall impact. Future efforts to implement decision aids should explore enhancing their integration into clinical workflows and standard operating procedures, supported by organizational incentives that prioritize shared decision-making. ClinicalTrials.gov NCT03985449; registered 6 June 2019.
- 5 November 2024
- Published by Springer Nature in Implementation Science
- Vol. 19 (1) , 1-13
Open AccessThe UPFRONT project: tailored implementation and evaluation of a patient decision aid to support shared decision-making about management of symptomatic uterine fibroids
To evaluate implementation of a patient decision aid for symptomatic uterine fibroid management to improve shared decision-making at five clinical settings across the United States. We used a type 3 hybrid effectiveness-implementation stepped-wedge design and the Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) planning and evaluation framework. We conducted clinician training, monthly reach tracking with feedback to site clinical leads, patient and clinician surveys, and visit audio-recordings. Implementation strategies included assessment of organizational readiness for shared decision-making, synchronous clinician training, audit and feedback of decision aid reach, and access to multiple decision aid formats. Outcomes and analyses included patient-level reach, clinician-level adoption, and associations of patient-reported decision aid exposure (as treated) and setting-level implementation (intention-to-treat) with patient-reported (collaboRATE measure) and observed (OPTION-5 measure) shared decision-making. We also designed and assessed setting-level plans for sustainability and other factors impacting sustained decision aid use. The decision aid was adopted by 72 of the 74 eligible gynecologists (97%) and reached 2553 patients across five settings. CollaboRATE scores improved among patients who reported receiving the decision aid (as-treated analysis, 69% vs. 59%; OR 1.6, 95% CI 1.16–2.27). CollaboRATE scores remained consistent before and after setting-level decision aid implementation (intention-to-treat analysis, 64% vs. 63%; OR 0.86, 95% CI 0.61–1.22). Participants would prefer to receive a decision aid at multiple time points (91.9% before the visit, 90.7% during the visit, 86.5% after the visit). Shared decision-making experiences did not improve when comparing pre vs. post-implementation collaboRATE scores across included settings (intention-to-treat, 64% vs. 63%; OR 0.86, 95% CI 0.61–1.22). When patients with symptomatic uterine fibroids are given decision aids, they report higher shared decision-making scores. However, the differences we observed between the as-treated and intention-to-treat results suggest that unaddressed implementation challenges continue to limit the extent to which patients receive decision aids and likely hinder their overall impact. Future efforts to implement decision aids should explore enhancing their integration into clinical workflows and standard operating procedures, supported by organizational incentives that prioritize shared decision-making. ClinicalTrials.gov NCT03985449; registered 6 June 2019.
- Open Access
- 16 September 2024
- Published by The Permanente Federation in The Permanente Journal
- Vol. 28 (3) , 278-283
Using Focus Groups to Evaluate a Self-Assessment Approach to Coproduction Value Creation in an International Community of Practice: Phase 2 of the CO-VALUE Study
Coproduction learning health system models clearly define the use of clinical and patient-reported data for system learning and quality improvement, but less is known about how to document formative learning about coproduction value creation over the course of a quality improvement initiative. The authors aimed to 1) assess the feasibility, utility, and acceptability of novel self-assessment tools for coproduction value creation and 2) identify domains of coproduction value creation. The authors conducted 4 focus groups with quality improvement teams from 4 health systems in the United States and Sweden between June 2021 and September 2023. A single analyst coded transcripts and proposed themes, with investigator triangulation validating results. Participants found the self-assessment tools acceptable and useful. The improvement passport was seen as more feasible for routine use than the full self-assessment guide. Peer learning within the community of practice, diverse multidisciplinary improvement teams, and leadership support facilitated teams' work. Domains of coproduction value creation included communication, self-efficacy, interconnectedness, direct and indirect costs of health care utilization, health professional experience, and access to the right care. Peer learning and camaraderie within the community of practice maintained momentum among participating teams during a challenging time of limited resources and mounting responsibilities in health care settings, suggesting enhanced resiliency through approaching difficult tasks in community. The authors identified themes of coproduction value creation and drivers of engagement. Future research will draw on the measurement domains established in this study to inform the development of measures of coproduction value creation. Those measures could then be incorporated into the data-rich environments of coproduction learning health systems to enhance focus on value from service user and professional perspectives.
- 16 September 2024
- Published by The Permanente Federation in The Permanente Journal
- Vol. 28 (3) , 278-283
Open AccessUsing Focus Groups to Evaluate a Self-Assessment Approach to Coproduction Value Creation in an International Community of Practice: Phase 2 of the CO-VALUE Study
Coproduction learning health system models clearly define the use of clinical and patient-reported data for system learning and quality improvement, but less is known about how to document formative learning about coproduction value creation over the course of a quality improvement initiative. The authors aimed to 1) assess the feasibility, utility, and acceptability of novel self-assessment tools for coproduction value creation and 2) identify domains of coproduction value creation. The authors conducted 4 focus groups with quality improvement teams from 4 health systems in the United States and Sweden between June 2021 and September 2023. A single analyst coded transcripts and proposed themes, with investigator triangulation validating results. Participants found the self-assessment tools acceptable and useful. The improvement passport was seen as more feasible for routine use than the full self-assessment guide. Peer learning within the community of practice, diverse multidisciplinary improvement teams, and leadership support facilitated teams' work. Domains of coproduction value creation included communication, self-efficacy, interconnectedness, direct and indirect costs of health care utilization, health professional experience, and access to the right care. Peer learning and camaraderie within the community of practice maintained momentum among participating teams during a challenging time of limited resources and mounting responsibilities in health care settings, suggesting enhanced resiliency through approaching difficult tasks in community. The authors identified themes of coproduction value creation and drivers of engagement. Future research will draw on the measurement domains established in this study to inform the development of measures of coproduction value creation. Those measures could then be incorporated into the data-rich environments of coproduction learning health systems to enhance focus on value from service user and professional perspectives.
- Open Access
- 26 February 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Using large language models to evaluate the offer of options in clinical encounters by focusing on an item of the Observer OPTION-5 measure of shared decision-making (Preprint)
UNSTRUCTURED Introduction: Human assessment of clinical encounter recordings using observer-based measures of shared decision-making, such as Observer OPTION-5 (OO5), is expensive. In this study, we aimed to assess the potential of using large language models (LLMs) to automate the rating of the OO5 item focused on offering options (item 1). Methods: We used a dataset of 287 clinical encounter transcripts of women diagnosed with early breast talking with their surgeon to discuss treatments. Each transcript had been previously scored by two researchers using OO5 (0 to 4 scale). We set up two rules-based baselines, one random and one using trigger words, and classified option talk instances using GPT-3.5 Turbo, GPT-4, and PaLM 2. To develop and compare the performance of these models, we randomly selected 16 transcripts for additional human annotation focusing on option talk instances (binary). To assess performance, we calculated Spearman correlations (rS) between the researcher-generated scores for item 1 for the remaining 271 transcripts and the item 1 instances predicted by the LLMs. Results: We observed high levels of correlation between the LLMs and researcher-generated scores. GPT-3.5 Turbo with a few-shot example had an rS=0.60 (P<.001) with the mean of the two scorers. Other LLMs had slightly lower correlation levels. Discussion: The LLMs, particularly GPT-3.5 Turbo with few-shot examples, demonstrated superior performance in identifying option talk instances compared to baseline models. GPT-3.5 Turbo demonstrated the best performance, achieving higher precision and recall. Conclusions: Further improvements in score correlations may be possible through improvements in and better understanding of LLMs.
- 26 February 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessUsing large language models to evaluate the offer of options in clinical encounters by focusing on an item of the Observer OPTION-5 measure of shared decision-making (Preprint)
UNSTRUCTURED Introduction: Human assessment of clinical encounter recordings using observer-based measures of shared decision-making, such as Observer OPTION-5 (OO5), is expensive. In this study, we aimed to assess the potential of using large language models (LLMs) to automate the rating of the OO5 item focused on offering options (item 1). Methods: We used a dataset of 287 clinical encounter transcripts of women diagnosed with early breast talking with their surgeon to discuss treatments. Each transcript had been previously scored by two researchers using OO5 (0 to 4 scale). We set up two rules-based baselines, one random and one using trigger words, and classified option talk instances using GPT-3.5 Turbo, GPT-4, and PaLM 2. To develop and compare the performance of these models, we randomly selected 16 transcripts for additional human annotation focusing on option talk instances (binary). To assess performance, we calculated Spearman correlations (rS) between the researcher-generated scores for item 1 for the remaining 271 transcripts and the item 1 instances predicted by the LLMs. Results: We observed high levels of correlation between the LLMs and researcher-generated scores. GPT-3.5 Turbo with a few-shot example had an rS=0.60 (P<.001) with the mean of the two scorers. Other LLMs had slightly lower correlation levels. Discussion: The LLMs, particularly GPT-3.5 Turbo with few-shot examples, demonstrated superior performance in identifying option talk instances compared to baseline models. GPT-3.5 Turbo demonstrated the best performance, achieving higher precision and recall. Conclusions: Further improvements in score correlations may be possible through improvements in and better understanding of LLMs.