Jessica Sousa

Large language model (LLM)-based chatbots are increasingly used for behavioral health support. Few studies have rigorously evaluated their advice on alcohol misuse. We evaluated seven publicly available chatbots-including general-purpose and behavioral health-focused tools-in responding to alcohol misuse-related questions. Using a fictional case, we simulated longitudinal chatbot interactions over seven days, using 25 prompts derived from real-world Reddit posts. Using an evaluation framework specific to chatbots, four clinicians independently rated each chatbot's transcript along five domains: empathy, quality of information, usefulness, responsiveness, and scope awareness. Clinicians also assessed secondary dimensions, including stigmatizing language and challenging the user (vs. only validating feelings). We generated descriptive statistics on performance and identified examples of problematic output. Across all chatbots, empathy was the highest-rated domain (mean score 4.6/5) while quality of information was the lowest (mean 2.7/5). There was considerable variation in overall mean performance scores across the chatbots, ranging from 2.1 (SD 1.1) to 4.5 (SD 0.8). There were no significant differences in performance between behavioral health and general-purpose chatbots. All chatbots had one or more examples of guidance deemed inappropriate, over-stated, or inaccurate. All avoided stigmatizing or judgmental language and supported self-efficacy. Chatbots were perceived to vary widely in their ability to support individuals with alcohol misuse. While generally strong in empathy, there is room for improvement in response quality. As chatbot use expands, users and clinicians should be aware of the strengths and weaknesses of chatbots in providing advice on alcohol misuse.

Although apps are widely available and have several advantages as a tool to support alcohol reduction and recovery, little is known about how individuals are using them. In 2024, we conducted an exploratory sequential mixed-methods study that coupled in-depth interviews with 22 app users and a nationally representative survey of 2002 adults. We explored experiences with and perceptions of alcohol reduction apps. Approximately 6% of US adults in the sample reported using alcohol reduction apps, and for most, it was the only support to address problematic drinking. In interviews, some users viewed apps as an alternative to traditional services and a way to independently address alcohol use; however, apps were seen as disconnected from care even by those who also used other supports. App users accessed a variety of features, with the most common being motivational content, tracking alcohol consumption, and educational content. Findings suggest that there are opportunities to not only introduce apps to individuals receiving healthcare services or participating in mutual support groups, but to reference and embed them in those settings. New approaches to regulation and reimbursement may support adoption as well as integration into healthcare services.

There is ongoing policy debate on the prescribing of controlled substances such as buprenorphine and stimulants via telemedicine. The goal of federal and state policymakers is to ensure access to care while limiting diversion risk. However, there is little evidence on how clinicians view and address diversion and on telemedicine's role in diversion. From December 2023 to January 2024, we conducted semi-structured interviews with 21 psychiatrists and primary care physicians engaged in hybrid (telemedicine and in-person) care models in which we explored perceptions of diversion and strategies used to monitor for diversion. Most physicians reported monitoring for diversion, but there was little consistency on how monitoring was done and reported strategies did not differ between telemedicine vs in-person care. When physicians suspected diversion, there was also wide variation in responses: some clinicians did not immediately take any action while others imposed more requirements on patients (e.g., more frequent visits), no longer prescribed the controlled substance, or terminated the patient from their practice. Few physicians had ever reported a case of suspected diversion to law enforcement. Our findings suggest that the Drug Enforcement Administration could clarify reporting requirements and professional societies could provide additional guidance on how to respond to suspected diversion, given the current variation in practice across clinicians could be exploited by individuals who want to divert.

The recent growth of telehealth may be impacting access to care for patients, including those with limited English proficiency (LEP). Using a secret-shopper design, simulated patients contacted 386 safety-net clinics in California in both Spanish and English from February–March 2023. Callers stated that they were new patients seeking medication for depression, and they documented time to an appointment and available visit modalities (telehealth and in-person). Multinomial logistic regression models examined associations between clinic characteristics and available modalities. English-speaking callers were more likely to speak with a live scheduler and to obtain appointment information from a scheduler who could engage with them in their preferred language. Among Spanish-speaking callers who reached a live scheduler, 22% reached someone who did not engage (eg, were hung up on) and, as a result, could not obtain appointment information. The mean estimated time to a prescribing visit was 36 days and did not differ by language. Sixty-four percent of clinics offered both telehealth and in-person visits, 14% only offered in-person visits, and 22% only offered telehealth visits. More attention and resources are needed to support patients with LEP at the point of scheduling and to ensure choice of visit modality for all patient populations.

Objective: During the first 9 months of the coronavirus disease 2019 (COVID-19) pandemic, many emergency departments (EDs) experimented with telehealth applications to reduce virus exposure, decrease visit volume, and conserve personal protective equipment. We interviewed ED leaders who implemented telehealth programs to inform responses to the ongoing COVID-19 pandemic and future emergencies. Materials and Methods: From September to November 2020, we conducted semi-structured interviews with ED leaders across the United States. We identified EDs with pandemic-related telehealth programs through literature review and snowball sampling. Maximum variation sampling was used to capture a range of experiences. We used standard qualitative analysis techniques, consisting of both inductive and deductive approaches to identify and characterize themes. Results: We completed 15 interviews with EDs leaders in 10 states. From March to November 2020, participants experimented with more than a dozen different types of telehealth applications including tele-isolation, tele-triage, tele-consultation, virtual postdischarge assessment, acute care in the home, and tele-palliative care. Prior experience with telehealth was key for implementation of new applications. Most new telehealth applications turned out to be temporary because they were no longer needed to support the response. The leading barriers to telehealth implementation during the pandemic included technology challenges and the need for “hands-on” implementation support in the ED. Conclusions: In response to the COVID-19 pandemic, EDs rapidly implemented many telehealth innovations. Their experiences can inform future responses.

RAND researchers describe practices identified by health centers participating in the Sustainable Models of Telehealth in the Safety Net initiative that could support the growth and sustainability of telemedicine programs in safety-net settings.

Objective: To evaluate changes in health and health care utilization outcomes for Latinx adults with substance use and mental disorders receiving integrated behavioral and primary health care. Design: Study sample included enrollees who completed baseline, 6-month and 12-month assessments (n = 107). Study outcomes were depression symptom severity, anxiety symptom severity, illicit drug use, emergency department utilization and homelessness status. Pre–post analyses were conducted using paired t-test and McNemar test to examine changes in study outcomes. Multivariable regression model estimated through generalized estimating equations explored the influence of the intervention on study outcomes. Results were presented in adjusted odds ratios (AOR) and 95% confidence intervals (CI). Results: Participants were less likely to report depressive symptoms (AOR: 0.496, 95%CI: 0.296–0.832), less likely to report anxiety symptoms (AOR: 0.539, 95%CI: 0.329–0.884), and less likely to experience homelessness (AOR: 0.556, 95%CI: 0.328–0.943) at 6-month assessment compared to baseline. Participants were also less likely to report depressive symptoms (AOR: 0.378, 95%CI: 0.209–0.684), less likely to report anxiety symptoms (AOR: 0.471, 95%CI: 0.270–0.821), less likely to experience homelessness (AOR: 0.333, 95%CI: 0.189–0.587), and less likely to utilize the emergency department in the past 30 days (AOR: 0.397, 95%CI: 0.188–0.837) at 12-month assessment compared to baseline. Conclusions: Integrating culturally responsive behavioral and primary health care services is critical for addressing the needs of Latinx adults with mental and substance use disorders, and other chronic diseases. This initiative has the potential to reduce disparities in access to and engagement in care for Latinx adults.