Carla A. Green

Purpose: To enhance automated methods for accurately identifying opioid‐related overdoses and classifying types of overdose using electronic health record (EHR) databases.Methods: We developed a natural language processing (NLP) software application to code clinical text documentation of overdose, including identification of intention for self‐harm, substances involved, substance abuse, and error in medication usage. Using datasets balanced with cases of suspected overdose and records of individuals at elevated risk for overdose, we developed and validated the application using Kaiser Permanente Northwest data, then tested portability of the application using Kaiser Permanente Washington data. Datasets were chart‐reviewed to provide a gold standard for comparison and evaluation of the automated method.Results: The method performed well in identifying overdose (sensitivity = 0.80, specificity = 0.93), intentional overdose (sensitivity = 0.81, specificity = 0.98), and involvement of opioids (excluding heroin, sensitivity = 0.72, specificity = 0.96) and heroin (sensitivity = 0.84, specificity = 1.0). The method performed poorly at identifying adverse drug reactions and overdose due to patient error and fairly at identifying substance abuse in opioid‐related unintentional overdose (sensitivity = 0.67, specificity = 0.96). Evaluation using validation datasets yielded significant reductions, in specificity and negative predictive values only, for many classifications mentioned above. However, these measures remained above 0.80, thus, performance observed during development was largely maintained during validation. Similar results were obtained when evaluating portability, although there was a significant reduction in sensitivity for unintentional overdose that was attributed to missing text clinical notes in the database.Conclusions: Methods that process text clinical notes show promise for improving accuracy and fidelity at identifying and classifying overdoses according to type using EHR data.

Purpose: To facilitate surveillance and evaluate interventions addressing opioid‐related overdoses, algorithms are needed for use in large health care databases to identify and differentiate community‐occurring opioid‐related overdoses from inpatient‐occurring opioid‐related overdose/oversedation.Methods: Data were from Kaiser Permanente Northwest (KPNW), a large integrated health plan. We iteratively developed and evaluated an algorithm for electronically identifying inpatient overdose/oversedation in KPNW hospitals from 1 January 2008 to 31 December 2014. Chart audits assessed accuracy; data sources included administrative and clinical records.Results: The best‐performing algorithm used these rules: (1) Include events with opioids administered in an inpatient setting (including emergency department/urgent care) followed by naloxone administration within 275 hours of continuous inpatient stay; (2) exclude events with electroconvulsive therapy procedure codes; and (3) exclude events in which an opioid was administered prior to hospital discharge and followed by readmission with subsequent naloxone administration. Using this algorithm, we identified 870 suspect inpatient overdose/oversedation events and chart audited a random sample of 235. Of the random sample, 185 (78.7%) were deemed overdoses/oversedation, 37 (15.5%) were not, and 13 (5.5%) were possible cases. The number of hours between time of opioid and naloxone administration did not affect algorithm accuracy. When “possible” overdoses/oversedations were included with confirmed events, overall positive predictive value (PPV) was very good (PPV = 84.0%). Additionally, PPV was reasonable when evaluated specifically for hospital stays with emergency/urgent care admissions (PPV = 77.0%) and excellent for elective surgery admissions (PPV = 97.0%).Conclusions: Algorithm performance was reasonable for identifying inpatient overdose/oversedation with best performance among elective surgery patients.

Purpose: The study aims to develop and validate algorithms to identify and classify opioid overdoses using claims and other coded data, and clinical text extracted from electronic health records using natural language processing (NLP).Methods: Primary data were derived from Kaiser Permanente Northwest (2008–2014), an integrated health care system (~n > 475 000 unique individuals per year). Data includedInternational Classification of Diseases, Ninth Revision(ICD‐9) codes for nonfatal diagnoses,International Classification of Diseases, Tenth Revision(ICD‐10) codes for fatal events, clinical notes, and prescription medication records. We assessed sensitivity, specificity, positive predictive value, and negative predictive value for algorithms relative to medical chart review and conducted assessments of algorithm portability in Kaiser Permanente Washington, Tennessee State Medicaid, and Optum.Results: Code‐based algorithm performance was excellent for opioid‐related overdoses (sensitivity = 97.2%, specificity = 84.6%) and classification of heroin‐involved overdoses (sensitivity = 91.8%, specificity = 99.0%). Performance was acceptable for code‐based suicide/suicide attempt classifications (sensitivity = 70.7%, specificity = 90.5%); sensitivity improved with NLP (sensitivity = 78.7%, specificity = 91.0%). Performance was acceptable for the code‐based substance abuse‐involved classification (sensitivity = 75.3%, specificity = 79.5%); sensitivity improved with the NLP‐enhanced algorithm (sensitivity = 80.5%, specificity = 76.3%). The opioid‐related overdose algorithm performed well across portability assessment sites, with sensitivity greater than 96% and specificity greater than 84%. Cross‐site sensitivity for heroin‐involved overdose was greater than 87%, specificity greater than or equal to 99%.Conclusions: Code‐based algorithms developed to detect opioid‐related overdoses and classify them according to heroin involvement perform well. Algorithms for classifying suicides/attempts and abuse‐related opioid overdoses perform adequately for use for research, particularly given the complexity of classifying such overdoses. The NLP‐enhanced algorithms for suicides/suicide attempts and abuse‐related overdoses perform significantly better than code‐based algorithms and are appropriate for use in settings that have data and capacity to use NLP.

Although many studies have documented patient-, clinician-, and organizational barriers/facilitators of primary care among people with mental illnesses, few have examined whether these factors predict actual rates of preventive service use. We assessed whether clinician behaviors, beliefs, characteristics, and clinician-reported organizational characteristics, predicted delivery of preventive services in this population. Primary care clinicians (n = 247) at Kaiser Permanente Northwest (KPNW) or community health centers and safety-net clinics (CHCs), in six states, completed clinician surveys in 2014. Using electronic health record data, we calculated preventive care-gap rates for patients with mental illnesses empaneled to survey respondents (n = 37,251). Using separate multi-level regression models for each setting, we tested whether survey responses predicted preventive service care-gap rates. After controlling for patient-level characteristics, patients of clinicians who reported a greater likelihood of providing preventive care to psychiatrically asymptomatic patients experienced lower care-gap rates (KPNW γ= − .05, p = .041; CHCs γ= − .05, p = .033). In KPNW, patients of female clinicians had fewer care gaps than patients of male clinicians (γ= − .07, p = .011). In CHCs, patients of clinicians who had practiced longer had fewer care gaps (γ= − .004, p = .010), as did patients whose clinicians believed that organizational quality goals facilitate preventive service provision (γ= − .06, p = .006). Case manager availability in CHCs was associated with higher care-gap rates (γ=.06, p = .028). Clinicians who report they are likely to address preventive concerns when their mentally ill patients present without apparent psychiatric symptoms had patients with fewer care gaps. In CHCs, care quality goals may facilitate preventive care whereas case managers may not.

A greater understanding of the factors that influence long-term sustainment of quality improvement (QI) initiatives is needed to promote organizational ability to sustain QI practices over time, help improve future interventions, and increase the value of QI investments. We approached 83 of 201 executive sponsors or change leaders at addiction treatment organizations that participated in the 2007–2009 NIATx200 QI intervention. We completed semi-structured interviews with 33 individuals between November 2015 and April 2016. NIATx200 goals were to decrease wait time, increase admissions and improve retention in treatment. Interviews sought to understand factors that either facilitated or impeded long-term sustainment of organizational QI practices made during the intervention. We used thematic analysis to organize the data and group patterns of responses. We assessed available quantitative outcome data and intervention engagement data to corroborate qualitative results. We used narrative analysis to group four important themes related to long-term sustainment of QI practices: (1) finding alignment between business- and client-centered practices; (2) staff engagement early in QI process added legitimacy which facilitated sustainment; (3) commitment to integrating data into monitoring practices and the identification of a data champion; and (4) adequate organizational human resources devoted to sustainment. We found four corollary factors among agencies which did not sustain practices: (1) lack of evidence of impact on business practices led to discontinuation; (2) disengaged staff and lack of organizational capacity during implementation period led to lack of sustainment; (3) no data integration into overall business practices and no identified data champion; and (4) high staff turnover. In addition, we found that many agencies’ current use of NIATx methods and tools suggested a legacy effect that might improve quality elsewhere, even absent overall sustainment of original study outcome goals. Available quantitative data on wait-time reduction demonstrated general concordance between agency perceptions of, and evidence for, sustainment 2 years following the end of the intervention. Additional quantitative data suggested that greater engagement during the intervention period showed some association with sustainment. Factors identified in QI frameworks as important for short-term sustainment—organizational capacity (e.g. staffing and leadership) and intervention characteristics (e.g. flexibility and fit)—are also important to long-term sustainment.

Purpose: The purpose of this study is to assess positive predictive value (PPV), relative to medical chart review, of International Classification of Diseases (ICD)‐9/10 diagnostic codes for identifying opioid overdoses and poisonings.Methods: Data were obtained from Kaiser Permanente Northwest and Northern California. Diagnostic data from electronic health records, submitted claims, and state death records from Oregon, Washington, and California were linked. Individual opioid‐related poisoning codes (e.g., 965.xx and X42), and adverse effects of opioids codes (e.g., E935.xx) combined with diagnoses possibly indicative of overdoses (e.g., respiratory depression), were evaluated by comparison with chart audits.Results: Opioid adverse effects codes had low PPV to detect overdoses (13.4%) as assessed in 127 charts and were not pursued. Instead, opioid poisoning codes were assessed in 2100 individuals who had those codes present in electronic health records in the period between the years 2008 and 2012. Of these, 10/2100 had no available information and 241/2100 were excluded potentially as anesthesia‐related. Among the 1849 remaining individuals with opioid poisoning codes, 1495 events were accurately identified as opioid overdoses; 69 were miscodes or misidentified, and 285 were opioid adverse effects, not overdoses. Thus, PPV was 81%. Opioid adverse effects or overdoses were accurately identified in 1780 of 1849 events (96.3%).Conclusions: Opioid poisoning codes have a predictive value of 81% to identify opioid overdoses, suggesting ICD opioid poisoning codes can be used to monitor overdose rates and evaluate interventions to reduce overdose. Further research to assess sensitivity, specificity, and negative predictive value are ongoing. Copyright © 2017 John Wiley & Sons, Ltd.