Marc N. Elliott

Objectives: The COVID-19 pandemic challenged hospitals. Yet, little is known about the extent of COVID-19’s impact on patient safety. We assessed change in Patient Safety Indicators (PSIs), which identify potentially preventable adverse events, before and after the onset of COVID-19 and when a hospital had a high proportion of COVID-19 admissions. Methods: Using Medicare Provider Analysis and Review data, our sample included 51,960,029 non-COVID-19 admissions eligible for inclusion in at least 1 PSI. This included 23,262,714 patients and a total of 294,871 PSI events. We fit risk-adjusted trends from 2018 to 2022 using interrupted time series models of PSIs before and after the onset of COVID-19. We also tested for the difference of a hospital having a high proportion of COVID-19 admissions. PSIs were categorized into provider-resource, surgical/team-resource, or hospital-resource-dependent events. Results: Ten (of 14) PSIs were worse after the onset of COVID-19 and had substantial increases in event rates, ranging from 14.6% (postoperative sepsis) to 51.9% (central venous catheter-related bloodstream infections). Also, when a hospital had a high proportion of COVID-19 admissions, 8 PSIs significantly worsened and had substantial increases in event rates, ranging from 5.9% (postoperative pulmonary embolism/deep venous thrombosis) to 15.8% (postoperative sepsis). Conclusion: The COVID-19 pandemic placed immense, prolonged strain on hospitals, resulting in a worsening of inpatient patient safety events. Increased rates of safety events occurred whether events were provider-resource, surgical/team-resource, or hospital-resource-dependent. These findings are a call for hospitals to return to placing heightened emphasis on reducing preventable harm and minimizing hospital complications and adverse events.

Background and Objectives: Many healthcare organizations administer Consumer Assessment of Healthcare Providers and Systems (CAHPS) surveys and use both quantitative ratings and qualitative narrative comments. While narrative data offer rich insights into patient experience, their time-intensive analysis often limits systematic use in quality improvement (QI). Healthcare organizations often underutilize this data, reading comments and not analyzing them systematically. This study examined the feasibility and accuracy of engaging clinicians in systematically coding narrative CAHPS data for QI. We conducted a coding exercise using patient-level narrative comments and compared the inter-rater reliability of clinicians and experienced qualitative researchers. Methods: We partnered with a large, urban freestanding children’s hospital that routinely collects Child HCAHPS data. Our six-person coding team comprised two experienced qualitative researchers, two pediatric inpatient physicians, and two pediatric inpatient nurses. Following standardized training on the coding process and a predefined codebook, the comments were coded independently. The coding framework included valence (positive, negative, mixed), identification of potential patient safety issues (safety flag), staff type mentioned (e.g., doctor, nurse), setting mentioned (e.g., emergency room), overall actionability (yes/no), and, if actionable, the level of action needed (individual clinician/provider or organizational). We calculated overall and pairwise inter-rater agreement (Kappa) to assess coding consistency across and between the three professional groups and for different code combinations to identify patterns. Results: Researchers had excellent inter-rater agreement (pooled Kappa = 0.88). There was very good agreement among clinicians (Kappa = 0.68), whereas nurses had good agreement (Kappa = 0.60). Valence and mentioned staff type were coded with high consistency across all groups. Agreement on the dichotomous 0/1 indicator of actionability (yes/no) was higher than for the three-level coding of actionability (not actionable/actionable at provider-level/actionable at organizational-level). Notably, nurses displayed slightly higher agreement than physicians, whereas researchers were consistently more concordant than both clinician groups. Nurses also demonstrated the strongest sensitivity in identifying potential patient safety issues within the narrative comments. Conclusion: Our findings suggest that training nurses to code narrative CAHPS comments for valence, mentioned staff, overall actionability, and patient safety flags is a feasible and potentially effective strategy for QI. This focused coding approach, implemented regularly (e.g., quarterly), could efficiently generate a substantial dataset for integration into routine QI initiatives and staff meetings.

Adults age 75+ report worse inpatient experiences than patients 55 to 74. Older adults may have been especially vulnerable to changes such as family access and reduced staffing during the COVID-19 pandemic. We examined 2018–2023 Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) surveys from 14,760,049 respondents discharged from 4,462 hospitals. We examined changes in an HCAHPS summary score (HCAHPS-SS), controlling for pre-pandemic quarterly and linear trends. HCAHPS-SS declined 4.1pp, a medium-to-large drop by Q3/2022, recovering only slightly by Q4/2023 (to −3.5pp). HCAHPS-SS decreased fastest for patients 75+ and least for maternity patients, even controlling for age. Differences by age may reflect older patients’ greater need for instrumental support and differential impacts of visitor restrictions. Hospitals’ quality improvement efforts should focus on understanding the need for patient support and on restoring prior patient experience gains. The failure to return to pre-pandemic levels points to the need to understand and address the residual factors that continue to alter patient experience.

Prior studies have found that critical access hospitals (CAHs), which serve patients who would otherwise have limited access to hospitals, provide lower-quality clinical care than inpatient prospective payment system (IPPS) hospitals; evidence is limited about the patient experiences they provide. Using linear mixed-effects regression models, we compared patient-mix–adjusted Hospital Consumer Assessment of Hospitals, Providers, and Systems (HCAHPS) survey scores for CAHs and IPPS hospitals and evaluated how much of the observed differences were associated with size, location, and other hospital characteristics. CAH patients were older, more often in the medical service line, had lower educational attainment, and worse self-rated health than their IPPS counterparts. Accounting for such differences, CAH patients had better experiences (+8 points on the 0–100 HCAHPS summary score, where differences >5 are considered large by patient experience heuristics), especially for staff responsiveness, cleanliness, quietness, and discharge information. CAHs do not outperform similarly small IPPS hospitals, which often have different missions (eg, for-profit surgical specialty hospitals). For-profit and teaching status, while uncommon among CAHs, predicted lower CAH HCAHPS performance. Despite the limited services provided by CAHs, their small scale may facilitate positive experiences for patients in areas with limited hospital choices. For-profit and teaching CAHs may benefit from quality-improvement efforts.

Importance: Influenza vaccination is a safe, effective way to mitigate influenza infection but remains underused by some groups, including older Black and Hispanic adults. There is concern that changing attitudes toward vaccination during the COVID-19 pandemic could decrease influenza vaccination rates.Objective: To examine national patterns of influenza vaccination among older adults with Medicare, including analyses stratified by race and ethnicity, educational attainment, and rurality.Design, Setting, and Participants: This retrospective, cross-sectional survey study of responses to the 2019 and 2022 Medicare Consumer Assessment of Healthcare Providers and Systems surveys used cross-sectional analysis for data from a weighted sample of 285 265 community-dwelling Medicare Advantage and Medicare Fee-for-Service enrollees aged 65 years or older living in the 50 US states and Washington, DC, who answered a survey question about influenza immunization.Exposures: Race and ethnicity, educational attainment, and rurality of survey respondents.Main Outcomes and Measures: The primary outcome was self-reported influenza vaccination.Results: The weighted sample combining both years of data for 285 265 individuals included 54.5% women. A total of 4.2% were Asian American, Native Hawaiian, and Other Pacific Islander; 8.0% were Black; 6.9% were Hispanic; and 76.2% were White. Between 2019 and 2022, overall influenza vaccination rates for older adults increased by 0.7 percentage points (95% CI, 0.2-1.1 percentage points), from 76.3% to 77.0%. Whereas influenza vaccination rates for Asian American, Native Hawaiian, and Other Pacific Islander older adults (from 81.2% to 83.1%) and White older adults (from 77.7% to 77.9%) were essentially unchanged between 2019 and 2022, they increased by 3.8 percentage points (95% CI, 1.8-5.7 percentage points) for Black older adults (from 66.9% to 70.7%) and 2.3 percentage points (95% CI, 0.5-4.0 percentage points) for Hispanic older adults (from 72.7% to 75.0%). Black and Hispanic older adults living in rural areas had the largest increases during this period (Black, 7.0 percentage points [95% CI, 0.3-13.8 percentage points]; Hispanic, 8.2 percentage points [95% CI, 0.8-15.5 percentage points]), while White older adults with lower educational attainment and/or living in rural areas had decreases in vaccination rates (White with lower educational attainment, –1.9 percentage points [95% CI, −2.8 to −1.0 percentage points]; White living in rural areas, –2.0 percentage points [95% CI, −3.2 to −0.8 percentage points]; White with lower educational attainment and living in rural areas, –3.7 percentage points [95% CI, −5.5 to −1.9 percentage points]).Conclusions and Relevance: In this cross-sectional survey study, although overall influenza vaccination rates changed little from 2019 to 2022, they increased substantially for Black and Hispanic older adults, particularly those in rural areas, and decreased for some groups of White older adults. Determining the reasons for these divergent changes in influenza vaccination rates is a high priority for future research.

Objectives: To present an example of using vignettes as an external anchor to assess measurement equivalence for patient experience measures. Background: Evaluating measurement equivalence and differences in scale use is helpful for identifying disparities in patient experience based on patient surveys. External anchors, often in the form of scored vignettes, provide an attractive approach to examining differences in scale use but are not commonly used. Methods: We analyzed a UK dataset based on the General Practice Patient Survey and a U.S. dataset based on the Consumer Assessment of Healthcare Providers and Systems Clinician and Group survey. A total of 560 White British and 560 Pakistani adults were recruited from various locations across England; 575 Asian American and 505 non-Hispanic White patients were recruited from an internet panel in the United States. Patient encounters and rated the quality of communication using 5 General Practice Patient Survey questions and 3 Consumer Assessment of Healthcare Providers and Systems Clinician and Group questions. Results: Using an external anchor in both United States and UK data produced substantial evidence of differential item functioning (DIF). However, an “internal” DIF analysis (without an external anchor) produced little evidence of DIF. Conclusions: Using an external anchor does not require the assumption made by internal methods that some items do not display between-group DIF. These assumptions may not hold for patient experience items if a single factor, such as extreme or negative response tendency, governs all items equally.

Background: Since its 2006 launch, the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey has become the most used hospital patient experience survey, included in public reporting and accountability systems. Responding to emerging evidence of possible gaps in survey content, the Centers for Medicare & Medicaid Services undertook extensive efforts to develop and test new HCAHPS measures. Objectives: Describe the development and testing of new HCAHPS measures. Research Design: Activities supporting the development and refinement of survey items and measures included 4 patient focus groups, 27 cognitive interviews, and a 12-member technical expert panel. Psychometric testing of proposed survey items and measures used data from a 2021 large-scale randomized experiment. Results: Patient focus groups, cognitive testing, psychometric analyses, and a technical expert panel supported the addition of 2 new multi-item measures (Care Coordination and Restfulness of Hospital Environment), a new single-item measure (Information about Symptoms), and the placement of one current HCAHPS item (Quietness) within one of the new composite measures. All proposed measures had strong psychometric properties, including good construct validity and hospital-level reliabilities that ranged from 0.73 for the Information about Symptoms standalone measure to 0.87 for Restfulness of Hospital Environment. Cronbach alpha for the 3 new or modified composite measures ranged from 0.74 to 0.76. Conclusions: The new content would broaden HCAHPS by adding new aspects of care that are important to patients. The public reporting of the new measures and their role in payment may support hospital quality improvement efforts.

Background: Web-first multimode survey protocols increase Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey response rates and representativeness but may result in different HCAHPS scores because of survey mode effects and selective email address availability. A variable absent from many patient-mix adjustment models that may result in more positive patient experiences is whether the hospital admission was planned; adjustment for planned stays may better measure hospital performance. Objectives: Develop adjustments for new Web-first survey protocols and planned admissions to facilitate comparisons across hospitals. Research Design: Using 2021 survey mode experiment data, we estimate survey protocol effects in linear models predicting HCAHPS top-box outcomes from protocol indicators (which incorporate email availability for Web-first protocols), patient-mix adjustors, and hospital intercepts. We evaluate the unique effect on scores of whether a stay was planned. Results: Phone-only and Web-Phone without email produce more positive responses than Mail-only, requiring negative adjustments. All other survey protocol effects and adjustments are mixed in direction and generally small. Planned stays are associated with more positive experiences for otherwise similar patients and make a unique contribution beyond other current patient-mix adjustment variables. Conclusions: It is important to adjust HCAHPS scores for survey protocol effects to ensure fair comparisons across hospitals and to enable hospitals to choose the survey protocol that best represents their patients. Incomplete email address availability necessitates that HCAHPS survey protocol adjustment control for email address availability when a Web-first protocol is used. Accounting for differences associated with planned stays may improve patient-mix adjustment.

Objectives: Patient experience surveys are essential to measuring patient-centered care, a key component of health care quality. Low response rates in underserved groups may limit their representation in overall measure performance and hamper efforts to assess health equity. Telephone follow-up improves response rates in many health care settings, yet little recent work has examined this for surveys of Medicare enrollees, including those with Medicare Advantage. Our objective was to describe response rates to the 2022 Medicare Consumer Assessment of Healthcare Providers and Systems (MCAHPS) surveys and the completion mode (mail or telephone), overall and by person-level characteristics. Study Design: Cross-sectional survey. Methods: Participants were 1,092,434 individuals with Medicare who were selected to receive the 2022 MCAHPS survey in the 50 states and the District of Columbia and who were representative of the Medicare population. Study measures were survey response and completion mode. Results: The overall response rate was 33.7% (31.3% by mail and 2.3% by telephone), with 6.9% of responses by telephone. Despite the low overall telephone response rate, the phone was used at markedly higher rates by respondents in some groups with lower overall response rates who are thus underrepresented among respondents, including those who were younger than 65 years (eligible for Medicare due to disability: 16.5% of responses by telephone), Black (16.1%), or Hispanic (14.1%) or had limited income and assets (14.6%). Conclusions: Including a telephone component in the administration of the MCAHPS survey continues to have value because several groups still show a relative preference for survey completion by telephone. Steps should be taken to improve response rates by telephone.