Tabor Elisabeth Flickinger

Effective patient-provider communication is crucial in clinical care, directly impacting patient outcomes and quality of life. Traditional evaluation methods, such as human ratings, patient feedback, and provider self-assessments, are often limited by high costs and scalability issues. Although existing natural language processing (NLP) techniques show promise, they struggle with the nuances of clinical communication and require sensitive clinical data for training, reducing their effectiveness in real-world applications. Emerging large language models (LLMs) offer a new approach to assessing complex communication metrics, with the potential to advance the field through integration into passive sensing and just-in-time intervention systems. This study explores LLMs as evaluators of palliative care communication quality, leveraging their linguistic, in-context learning, and reasoning capabilities. Specifically, using simulated scripts crafted and labeled by healthcare professionals, we test proprietary models (e.g., GPT-4) and fine-tune open-source LLMs (e.g., LLaMA2) with a synthetic dataset generated by GPT-4 to evaluate clinical conversations, to identify key metrics such as “understanding” and “empathy.” Our findings demonstrated LLMs’ superior performance in evaluating clinical communication, providing actionable feedback with reasoning, and demonstrating the feasibility and practical viability of developing in-house LLMs. This research highlights LLMs’ potential to enhance patient-provider interactions and lays the groundwork for downstream steps in developing LLM-empowered clinical health systems.

Background: Quality patient-clinician communication is critical to ensure optimal patient outcomes. Recent advancements in artificial intelligence (AI) have increased the potential to develop technologies to evaluate and support patient-clinician communication. One example is CommSense, an AI-supported communication technology designed by our team to record and analyze clinical conversations and provide clinicians with actionable, tailored feedback to enhance communication performance. Objective: To explore system-level considerations to successfully implement communication technologies into healthcare contexts, using CommSense as an example case. Methods: This descriptive qualitative study used a purposive sampling strategy to recruit participants with communication expertise across four categories: administrators; industry; policymakers; and scientists. Semi-structured interviews focused on: 1) general barriers to patient-clinician communication; and 2) implementation considerations for integrating communication technologies into healthcare systems. Interviews were audio-recorded, transcribed verbatim, verified, and then analyzed using a combination of inductive and deductive coding strategies to identify broader patterns and themes. Results: Participants (n=13) discussed general barriers to quality communication between patients and clinicians, including: inadequate clinician communication training; generalized versus tailored patient communication; and hesitancy to initiate conversations with patients regarding difficult or sensitive issues, such as advance care planning. System-level implementation considerations focused on: the importance of integrating AI-supported communication technologies within existing health system infrastructure, clinical workflows, and policies and procedures; engagement of champions throughout the implementation process; and intentionality with the design and delivery of communication performance feedback. Participants also emphasized the particular benefit of communication technologies to support trainees and their learning. Conclusion: AI-supported communication technologies have the potential to address persistent barriers to quality communication within healthcare. Our findings demonstrate the importance of including decision-makers and outside experts across multiple domains to ensure successful implementation of AI-supported communication tools within complex healthcare systems.

Background: Evaluating implementation of digital health interventions (DHIs) in practice settings is complex, involving diverse users and multistep processes. Proactive planning can ensure implementation determinants and outcomes are captured for hybrid studies, but operational guidance for designing or planning hybrid DHI studies is limited. Objective: This study aimed to proactively define, prioritize, and operationalize measurement of implementation outcomes and determinants for a DHI hybrid effectiveness-implementation trial. We describe unique advantages and limitations of planning the trial implementation evaluation among a large-scale cohort study population and share results of a pretrial organizational readiness assessment. Methods: We planned a cluster-randomized, type II hybrid effectiveness-implementation trial testing PositiveLinks , a smartphone app for HIV care, compared to usual care (n=6 sites per arm), among HIV outpatient sites in the DC Cohort Longitudinal HIV Study in Washington, DC. We (1) defined components of the DHI and associated implementation strategy; (2) selected implementation science frameworks to accomplish evaluation aims; (3) mapped framework dimensions, domains, and constructs to implementation strategy steps; (4) modified or created instruments to collect data for implementation outcome measures and determinants; and (5) developed a compatible implementation science data collection and management plan. Provider baseline surveys administered at intervention sites probed usage of digital tools and assessed provider readiness for implementation with the Organizational Readiness to Implement Change tool. Results: We specified DHI and implementation strategy toward planning measurement of DHI and broader program reach and adoption. Mapping of implementation strategy steps to the Reach Effectiveness Adoption Implementation Maintenance framework prompted considerations for how to capture understudied aspects of each dimension: denominators and demographic representativeness within reach or adoption, and provider or organization-level adaptations, dose, and fidelity within the implementation dimension. Our process also prompted the creation of tools to obtain detailed determinants across domains and constructs of the Consolidated Framework for Implementation Research within a large sample at multiple time points. Some aspects of real-world PositiveLinks implementation were not reflected within the planned hybrid trial (eg, research assistants selected as de facto site implementation leads) or were modified to preserve internal validity of effectiveness measurement (eg, “Community of Practice”). Providers and research assistants (n=17) at intervention sites self-reported high baseline use of digital tools to communicate with patients. Readiness assessment revealed high median (48, IQR 45‐54) total Organizational Readiness to Implement Change scores, with research assistants scoring higher than physicians (52.5, IQR 44-55 vs 48.0, IQR 46-49). Conclusions: Key takeaways, challenges, and opportunities arose in planning the implementation evaluation within a hybrid DHI trial among a cohort population. Prospective trial planning must balance generalizability of implementation processes to “real world” conditions with rigorous procedures to measure intervention effectiveness. Rapid, scalable tools require further study to enable evaluations within large multisite hybrid studies.

BACKGROUND Evaluating implementation of digital health interventions (DHIs) in practice settings is complex, involving diverse users and multistep processes. Proactive planning can ensure implementation determinants and outcomes are captured for hybrid studies, but operational guidance for DHI studies is limited. OBJECTIVE We aimed to proactively define, prioritize and operationalize measurement of implementation outcomes and determinants for a DHI hybrid effectiveness-implementation trial. We describe unique advantages and limitations of planning the trial among a large-scale cohort study population and share results of a pre-trial organizational readiness assessment. METHODS We planned a cluster randomized, type II hybrid effectiveness-implementation trial testing PositiveLinks, a smartphone application for HIV care, compared to usual care (n=6 sites per arm), among HIV outpatient sites in the DC Cohort Longitudinal HIV Study in Washington, DC. We 1) Defined components of the DHI and associated implementation strategy, 2) Selected implementation science frameworks to accomplish evaluation aims, 3) Mapped framework dimensions, domains, and constructs to implementation strategy steps, 4) Modified or created instruments to collect data for implementation outcome measures and determinants and 5) Developed a compatible implementation science data collection and management plan. Provider baseline surveys administered at intervention sites probed usage of digital tools and assessed provider readiness for implementation with the Organizational Readiness to Implement Change (ORIC) tool. RESULTS Specification of components of the DHI and implementation strategy identified relevant platform usage data and necessary implementer actions, toward planning measurement of program reach and adoption. A priori mapping of implementation strategy steps to the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework identified how data would be collected for each step/corresponding outcome measure. The multi-site cohort study provided infrastructure for prospective, scaled implementation research, with site research assistants (RAs) coordinating implementation. Existing cohort study tools (periodic site assessments, patient consent logs) were adapted for this evaluation to capture representativeness and reach/adoption ‘denominators.’ New survey instruments allowed for framework-guided evaluation of provider adaptations, dose, and fidelity to planned implementation across more participants and multiple timepoints. Some aspects of real-world PositiveLinks implementation were not reflected within the planned hybrid trial (e.g. RAs selected as de facto site implementation leads) or were modified to preserve internal validity of effectiveness measurement (e.g. ‘Community of Practice’). Providers and RAs (n=17) at intervention sites self-reported high baseline usage of digital tools to communicate with patients. Readiness assessment revealed high median total ORIC scores (48; IQR: 45-54), with RAs scoring higher than physicians (52.5 vs 48.0). CONCLUSIONS Challenges and opportunities arose in planning the implementation evaluation within a hybrid trial among a cohort population. Prospective hybrid trial planning must balance generalizability of implementation processes to ‘real world’ conditions with rigorous procedures to measure intervention effectiveness. Rapid, scalable tools require further study to enable evaluations within large multi-site hybrid studies. CLINICALTRIAL NCT04998019 INTERNATIONAL REGISTERED REPORT RR2-10.2196/37748

Large language models (LLMs) are transforming healthcare by advancing clinical decision support, patient care, and administrative efficiency. However, effectively and sustainably integrating LLMs into healthcare systems requires addressing participatory gaps that may hinder alignment with stakeholders' practical and ethical needs. This paper explores how participatory methods can be applied throughout the development lifecycle of LLM-enhanced health systems (LLMHS), arguing that: 1) participatory approaches are critical for engaging stakeholders in LLMHS development, and 2) LLM techniques can create novel participatory opportunities that reinforce stakeholder engagement while driving technical innovation in LLMHS. This dual perspective highlights the potential of LLMHS to align technical sophistication with real-world healthcare demands, paving the way for next-generation health systems.

By consistently taking medication, people with HIV (PWH) can attain viral suppression, improving their health and reducing transmission risk. PositiveLinks (PL) is a clinic-deployed mobile platform designed to improve engagement in care for PWH by enabling them to track their medications, connect with peers, and communicate with providers. This project investigated the experience of PL users who had recent periods of viral non-suppression to understand how these high-risk episodes can be predicted and prevented. Fifteen participants completed mental health measures and in-depth interviews. Two members of the study team independently coded interview transcripts and resolved any discrepancies. The codebook was developed iteratively until thematic saturation and intercoder reliability were achieved. Participants revealed mental health and substance use concerns. Frequent stressors highlighted were relationship challenges, financial difficulties, and unstable living situations. Relationship challenges were a common barrier to care, along with competing priorities, transportation and finances. In contrast, positive relationships, motivation, positive interactions with clinic staff, and social programs facilitated regaining viral suppression. Participants highlighted the importance of strong patient-provider relationships and social support in overcoming barriers to care. We identified factors that impact medication adherence, which will inform efforts to mitigate and prevent viral non-suppression and thus improve health outcomes.

Quality patient-provider communication is critical to improve clinical care and patient outcomes. While progress has been made with communication skills training for clinicians, significant gaps exist in how to best monitor, measure, and evaluate the implementation of communication skills in the actual clinical setting. Advancements in ubiquitous technology and natural language processing make it possible to realize more objective, real-time assessment of clinical interactions and in turn provide more timely feedback to clinicians about their communication effectiveness. In this paper, we propose CommSense, a computational sensing framework that combines smartwatch audio and transcripts with natural language processing methods to measure selected "best-practice'' communication metrics captured by wearable devices in the context of palliative care interactions, including understanding, empathy, presence, emotion, and clarity. We conducted a pilot study involving N=40 clinician participants, to test the technical feasibility and acceptability of CommSense in a simulated clinical setting. Our findings demonstrate that CommSense effectively captures most communication metrics and is well-received by both practicing clinicians and student trainees. Our study also highlights the potential for digital technology to enhance communication skills training for healthcare providers and students, ultimately resulting in more equitable delivery of healthcare and accessible, lower cost tools for training with the potential to improve patient outcomes.

Quality patient-provider communication is critical to improve clinical care and patient outcomes. While progress has been made with communication skills training for clinicians, significant gaps exist in how to best monitor, measure, and evaluate the implementation of communication skills in the actual clinical setting. Advancements in ubiquitous technology and natural language processing make it possible to realize more objective, real-time assessment of clinical interactions and in turn provide more timely feedback to clinicians about their communication effectiveness. In this paper, we propose CommSense, a computational sensing framework that combines smartwatch audio and transcripts with natural language processing methods to measure selected ``best-practice'' communication metrics captured by wearable devices in the context of palliative care interactions, including understanding, empathy, presence, emotion, and clarity. We conducted a pilot study involving N=40 clinician participants, to test the technical feasibility and acceptability of CommSense in a simulated clinical setting. Our findings demonstrate that CommSense effectively captures most communication metrics and is well-received by both practicing clinicians and student trainees. Our study also highlights the potential for digital technology to enhance communication skills training for healthcare providers and students, ultimately resulting in more equitable delivery of healthcare and accessible, lower cost tools for training with the potential to improve patient outcomes.

In HIV care, a strong rapport between patient and provider is essential for strengthening trust, enhancing therapy adherence, and ultimately leading to improved health outcomes. As the adoption of digital interactions in HIV care via mobile health (mHealth) tools is emerging, maintaining rapport in these asynchronous text-based communications becomes a critical yet challenging task. In this paper, we analyze 1,740 messages from an mHealth platform, categorized by experienced clinicians as either ‘rapport-building’ or ‘information-only.’ We utilize linguistic analysis to uncover key attributes of rapport-building communication. This led to a set of machine learning (ML) models and Large Language Models (LLMs) capable of classifying these communication styles. Further, we propose the application of LLMs not only to identify but also to actively rewrite ‘information only’ messages into versions that enhance rapport building without compromising information integrity. Our research demonstrates potential advancements in HIV mHealth communication by integrating linguistic analysis with language models, leading to more effective patient-provider interactions.

Background: HIV viral suppression and retention in care continue to be challenging goals for people with HIV in Washington, District of Columbia (DC). The PositiveLinks mobile app is associated with increased retention in care and viral load suppression in nonurban settings. The app includes features such as daily medication reminders, mood and stress check-ins, an anonymized community board for peer-to-peer social support, secure messaging to care teams, and resources for general and clinic-specific information, among other features. PositiveLinks has not been tailored or tested for this distinct urban population of people with HIV. Objective: This study aimed to inform the tailoring of a mobile health app to the needs of people with HIV and their providers in Washington, DC. Methods: We conducted a 3-part formative study to guide the tailoring of PositiveLinks for patients in the DC Cohort, a longitudinal cohort of >12,000 people with HIV receiving care in Washington, DC. The study included in-depth interviews with providers (n=28) at study clinics, focus groups with people with HIV enrolled in the DC Cohort (n=32), and a focus group with members of the DC Regional Planning Commission on Health and HIV (COHAH; n=35). Qualitative analysis used a constant comparison iterative approach; thematic saturation and intercoder agreement were achieved. Emerging themes were identified and grouped to inform an adaptation of PositiveLinks tailored for patients and providers. Results: Emerging themes for patients, clinic providers, and COHAH providers included population needs and concerns, facilitators and barriers to engagement in care and viral suppression, technology use, anticipated benefits, questions and concerns, and suggestions. DC Cohort clinic and COHAH provider interviews generated an additional theme: clinic processes. For patients, the most commonly discussed potential benefits included improved health knowledge and literacy (mentioned n=10 times), self-monitoring (n=7 times), and connection to peers (n=6 times). For providers, the most common anticipated benefits were improved communication with the clinic team (n=21), connection to peers (n=14), and facilitation of self-monitoring (n=11). Following data review, site principal investigators selected core PositiveLinks features, including daily medication adherence, mood and stress check-ins, resources, frequently asked questions, and the community board. Principal investigators wanted English and Spanish versions depending on the site. Two additional app features (messaging and documents) were selected as optional for each clinic site. Overall, 3 features were not deployed as not all participating clinics supported them. Conclusions: Patient and provider perspectives of PositiveLinks had some overlap, but some themes were unique to each group. Beta testing of the tailored app was conducted (August 2022). This formative work prepared the team for a cluster randomized controlled trial of PositiveLinks’ efficacy. Randomization of clinics to PositiveLinks or usual care occurred in August 2022, and the randomized controlled trial launched in November 2022. International Registered Report Identifier (IRRID): RR2-10.2196/37748