Karin Md Nelson

Ambient artificial intelligence (AI) scribes can reduce the burden of administrative documentation. Prior evaluations have been vendor specific and not focused on measures of documentation quality. To compare the quality of AI-generated clinical notes with that of human-produced notes. Cross-sectional evaluation of notes generated from standardized primary care clinical cases. Veterans Health Administration (VHA). 11 AI scribe tools, 18 human note takers, and 30 human raters. Five standardized primary care cases were audio recorded using standardized patients (for example, new patient, back pain, chest pain, pharmacy, and nurse care manager). Vendors and human clinicians generated encounter notes from the audio files. Blinded raters assessed all notes using the modified Physician Documentation Quality Instrument (PDQI-9), which measures 10 domains of note quality on a 5-point Likert scale (maximum score 50). Across all 5 clinical cases, human-generated notes received higher overall modified PDQI-9 scores than AI-generated notes. The largest difference was seen in the acute low back pain case (human: 43.8 [95% CI, 37.4 to 50.3] vs. AI: 20.3 [CI, 15.4 to 25.2]; difference -23.5 [CI, -29.2 to -17.9]). Pooled domain analysis showed lower AI scores across all 10 domains, with the largest deficits in domains related to being thorough (-1.23 [CI, -1.82 to -0.65]), organized (-1.06 [CI, -1.65 to -0.47]), and useful (-1.03 [CI, -1.61 to -0.44]). Cases were simulated; human-generated notes were not generated under real-world constraints. Notes generated by AI had lower-quality scores than human-generated notes across 5 standardized care cases. Although ambient AI scribes hold promise for reducing clinician burden, independent, vendor-neutral evaluations of note quality are essential before large-scale clinical deployment. VHA.

Background: Prior research has shown that primary care clinicians (PCPs) spend a large portion of clinic visits on tasks within the electronic health record (EHR). However, no time allocation studies have been done in the Veterans Health Administration (VHA) and little is known about EHR time spent during virtual visits. Objective: To estimate the proportion of clinician time spent working within the EHR during primary care visits at VHA clinics. Design: We used a time study software (WorkStudy+) adapted from prior research to collect time-motion data via remote observation of in-person and virtual visits. Participants: 23 PCPs (including physicians and nurse practitioners) from 3 regional VA sites. Main Measures: Proportion of observed time spent interfacing with the EHR during a primary care visit. Key Results: Of 211 primary care visits observed, the average visit length was 23.9 min in-person, 21.2 min by phone, and 29.3 min on video. The percentage of time spent on EHR work during the visit was 35% for in-person visits, 46% for phone visits, and 39% for video visits. During n=39 4-h clinic sessions, PCPs spent 54 min completing administrative tasks between patient visits, with 44% of time spent on documentation, 14% on chart review, and 14% on placing orders. Conclusions: PCPs at the VHA spend between one-third and one-half of each patient visit interfacing with the EHR. Most of this time is spent on documentation and chart review. Less time was spent in the EHR during in-person visits compared to virtual visits, suggesting that clinicians limit EHR task completion when the patient is present. Between patient visits during clinic sessions, PCPs spend 75% of their time working in the EHR. In total, this represents over 2 h per half-day clinic session spent on EHR tasks.

Objectives: We examined how individual-level turnover among Veterans Health Administration primary care providers (PCPs) from fiscal years 2017 to 2021 was associated with health care system-level burnout and turnover intent. Background: Burnout among PCPs has been well documented in recent studies, but less is known about the potential relationship between burnout and turnover. Methods: We identified a national cohort of 6444 PCPs (physicians, nurse practitioners, and physician assistants) in 129 Veterans Health Administration health care systems in the first quarter of fiscal year 2017 and tracked their employment status for 20 quarters. PCP employment data on turnover were linked to annual health care system-level employee survey data on burnout, turnover intent, and other covariates. We performed logistic regression to estimate the impact of health care system-level burnout and turnover intent on individual PCP turnover, controlling for individual and health care system-level covariates and adjusting for clustering at the health care system level. Results: Median health care system-level burnout ranged from 42.5% to 52.0% annually, and turnover among PCPs ranged from 6.3% to 8.4% (mean = 7.0%; SD = 0.9%). Separation from employment was higher among employees at health care systems with the highest burnout (odds ratio =1.14; 95% CI = 1.01–1.29) and turnover intent (OR = 1.18; 95% CI = 1.03–1.35). Conclusions: PCPs in health care systems with high burnout are more likely to separate from employment. Policymakers and administrators seeking to improve retention should consider system-level interventions to address organizational drivers of burnout.

Objective: To evaluate whether the Preventive Health Inventory (PHI)—a virtual care management intervention addressing hypertension and diabetes management implemented nationally in the Veterans Health Administration (VHA)—was delivered equitably among racial/ethnic groups and if existing inequities in hypertension and diabetes outcomes changed following PHI receipt.Data Sources and Study Setting: We used data from the VHA Corporate Data Warehouse among Veterans enrolled in primary care nationally from February 28, 2021 to March 31, 2022.Study Design: We used logistic regression to evaluate PHI receipt and hypertension and diabetes outcomes after PHI implementation among Veterans with hypertension and/or diabetes. We conducted unadjusted analyses and analyses adjusting for clinic fixed effects using dummy variables.Data Collection/Extraction Methods: We identified Veterans engaged in primary care with documented race/ethnicity and hypertension and/or diabetes diagnoses in all months during the study period.Principle Findings: Prior to PHI, Non‐Hispanic Black (NHB) (42.2%) and Hispanic (39.5%) Veterans were less likely to have controlled hypertension vs. Non‐Hispanic White (NHW) Veterans (47.5%); NHB Veterans (32.9%) were more likely to have uncontrolled diabetes vs. NHW Veterans (25.1%). Among 1,805,658 Veterans, 5.7% NHW (N = 68,744), 5.6% NHB (N = 22,580), 10.2% Hispanic (N = 13,313), 6.2% Asian/Pacific Islander/Native Hawaiian (N = 1868), 5.1% American Indian/Native Alaskan (N = 744), and 5.6% multiple races or other race (N = 1647) Veterans received PHI. We found no significant racial inequities in PHI receipt in unadjusted and adjusted models. Hypertension and diabetes measures improved more in the intervention group compared with the group who did not receive the intervention. There were no new or worsened inequities after PHI, and in pre‐/post‐intervention analysis, among NHB Veterans, the inequity in uncontrolled diabetes improved by 1.9 percentage points (95% CI 0.2, 3.6).Conclusions: Our findings suggest the PHI intervention was equitably deployed across race/ethnicity groups without significantly impacting most existing inequities in diabetes and hypertension.

Background: The Veterans Affairs (VA) Clinical Resource Hub (CRH) program aims to improve patient access to care by implementing time-limited, regionally based primary or mental health staffing support to cover local staffing vacancies. VA’s Office of Primary Care (OPC) designed CRH to support more than 1000 geographically disparate VA outpatient sites, many of which are in rural areas, by providing virtual contingency clinical staffing for sites experiencing primary care and mental health staffing deficits. The subsequently funded CRH evaluation, carried out by the VA Primary Care Analytics Team (PCAT), partnered with CRH program leaders and evaluation stakeholders to develop a protocol for a six-year CRH evaluation. The objectives for developing the CRH evaluation protocol were to prospectively: 1) identify the outcomes CRH aimed to achieve, and the key program elements designed to achieve them; 2) specify evaluation designs and data collection approaches for assessing CRH progress and success; and 3) guide the activities of five geographically dispersed evaluation teams. Methods: The protocol documents a multi-method CRH program evaluation design with qualitative and quantitative elements. The evaluation’s overall goal is to assess CRH’s return on investment to the VA and Veterans at six years through synthesis of findings on program effectiveness. The evaluation includes both observational and quasi-experimental elements reflecting impacts at the national, regional, outpatient site, and patient levels. The protocol is based on program evaluation theory, implementation science frameworks, literature on contingency staffing, and iterative review and revision by both research and clinical operations partners. Discussion: Health systems increasingly seek to use data to guide management and decision-making for newly implemented clinical programs and policies. Approaches for planning evaluations to accomplish this goal, however, are not well-established. By publishing the protocol, we aim to increase the validity and usefulness of subsequent evaluation findings. We also aim to provide an example of a program evaluation protocol developed within a learning health systems partnership.

Background: The COVID-19 pandemic has amplified the need for web-based behavioral interventions to support individuals who are diagnosed with chronic conditions and their informal caregivers. However, most interventions focus on patient outcomes. Dyadic technology–enabled interventions that simultaneously improve outcomes for patients and caregivers are needed. Objective: This study aimed to describe the methodology used to adapt a telephone-based, facilitated, and dyadic self-management program called Self-care Using Collaborative Coping Enhancement in Diseases (SUCCEED) into a self-guided, web-based version (web-SUCCEED) and to conduct usability testing for web-SUCCEED. Methods: We developed web-SUCCEED in 6 steps: ideation—determine the intervention content areas; prototyping—develop the wireframes, illustrating the look and feel of the website; prototype refinement via feedback from focus groups; finalizing the module content; programming web-SUCCEED; and usability testing. A diverse team of stakeholders including content experts, web designers, patients, and caregivers provided input at various stages of development. Costs, including full-time equivalent employee, were summarized. Results: At the ideation stage, we determined the content of web-SUCCEED based on feedback from the program’s original pilot study. At the prototyping stage, the principal investigator and web designers iteratively developed prototypes that included inclusive design elements (eg, large font size). Feedback about these prototypes was elicited through 2 focus groups of veterans with chronic conditions (n=13). Rapid thematic analysis identified two themes: (1) web-based interventions can be useful for many but should include ways to connect with other users and (2) prototypes were sufficient to elicit feedback about the esthetics, but a live website allowing for continual feedback and updating would be better. Focus group feedback was incorporated into building a functional website. In parallel, the content experts worked in small groups to adapt SUCCEED’s content, so that it could be delivered in a didactic, self-guided format. Usability testing was completed by veterans (8/16, 50%) and caregivers (8/16, 50%). Veterans and caregivers gave web-SUCCEED high usability scores, noting that it was easy to understand, easy to use, and not overly burdensome. Notable negative feedback included “slightly agreeing” that the site was confusing and awkward. All veterans (8/8, 100%) agreed that they would choose this type of program in the future to access an intervention that aims to improve their health. Developing and maintaining the software and hosting together cost approximately US $100,000, excluding salary and fringe benefits for project personnel (steps 1-3: US $25,000; steps 4-6: US $75,000). Conclusions: Adapting an existing, facilitated self-management support program for delivery via the web is feasible, and such programs can remotely deliver content. Input from a multidisciplinary team of experts and stakeholders can ensure the program’s success. Those interested in adapting programs should have a realistic estimate of the budget and staffing requirements.

Importance: Although cardiovascular disease (CVD) is the leading cause of death in the US, CVD risk factors remain suboptimally controlled.Objective: To test the effectiveness of a home-visit, peer health coaching intervention to improve health outcomes for veterans with multiple CVD risks.Design, Setting, and Participants: This 2-group, unblinded randomized clinical trial, called Vet-COACH (Veteran Peer Coaches Optimizing and Advancing Cardiac Health), used a novel geographic-based method to recruit a racially diverse population of veterans with low income. These veterans were enrolled at the Seattle or American Lake Veterans Health Affairs primary care clinics in Washington state. Veterans with a diagnosis of hypertension with at least 1 blood pressure reading of 150/90 mm Hg or higher in the past year, and 1 other CVD risk factor (current smoker, overweight or obesity, and/or hyperlipidemia), who resided in Census tracts with the highest prevalence of hypertension were eligible to participate. Participants were randomized to the intervention group (n = 134) or control group (n = 130). An intention-to-treat analysis was performed from May 2017 to October 2021.Intervention: Participants in the intervention group received peer health coaching for 12 months with mandatory and optional educational materials, an automatic blood pressure monitor, a scale, a pill organizer, and healthy nutrition tools. Participants in the control group received usual care plus educational materials.Main Outcomes and Measures: The primary outcome was a change in systolic blood pressure (SBP) from baseline to 12-month follow-up. Secondary outcomes included change in health-related quality of life (HRQOL; measured using the 12-item Short Form survey’s Mental Component Summary and Physical Component Summary scores), Framingham Risk Score, and overall CVD risk and health care use (hospitalizations, emergency department visits, and outpatient visits).Results: The 264 participants who were randomized (mean [SD] age of 60.6 [9.7] years) were predominantly male (229 [87%]) and 73 (28%) were Black individuals and 103 (44%) reported low annual income (<$40 000 per year). Seven peer health coaches were recruited. No difference was found in change in SBP between the intervention and control groups (−3.32 [95% CI, −6.88 to 0.23] mm Hg vs −0.40 [95% CI, −4.20 to 3.39] mm Hg; adjusted difference in differences, −2.05 [95% CI, −7.00 to 2.55] mm Hg; P = .40). Participants in the intervention vs control group reported greater improvements in mental HRQOL scores (2.19 [95% CI, 0.26-4.12] points vs −1.01 [95% CI, −2.91 to 0.88] points; adjusted difference in differences, 3.64 [95% CI, 0.66-6.63] points; P = .02). No difference was found in physical HRQOL scores, Framingham Risk Scores, and overall CVD risk or health care use.Conclusions and Relevance: This trial found that, although the peer health coaching program did not significantly decrease SBP, participants who received the intervention reported better mental HRQOL compared with the control group. The results suggest that a peer-support model that is integrated into primary care can create opportunities for well-being improvements beyond blood pressure control.Trial Registration: ClinicalTrials.gov Identifier: NCT02697422