Comparison of “Risk-Adjusted” Hospital Outcomes
Open Access
- 15 April 2008
- journal article
- research article
- Published by Wolters Kluwer Health in Circulation
- Vol. 117 (15) , 1955-1963
Abstract
Background— A frequent challenge in outcomes research is the comparison of rates from different populations. One common example with substantial health policy implications involves the determination and comparison of hospital outcomes. The concept of “risk-adjusted” outcomes is frequently misunderstood, particularly when it is used to justify the direct comparison of performance at 2 specific institutions. Methods and Results— Data from 14 Massachusetts hospitals were analyzed for 4393 adults undergoing isolated coronary artery bypass graft surgery in 2003. Mortality estimates were adjusted using clinical data prospectively collected by hospital personnel and submitted to a data coordinating center designated by the state. The primary outcome was hospital-specific, risk-standardized, 30-day all-cause mortality after surgery. Propensity scores were used to assess the comparability of case mix (covariate balance) for each Massachusetts hospital relative to the pool of patients undergoing coronary artery bypass grafting surgery at the remaining hospitals and for selected pairwise comparisons. Using hierarchical logistic regression, we indirectly standardized the mortality rate of each hospital using its expected rate. Predictive cross-validation was used to avoid underidentification of true outlying hospitals. Overall, there was sufficient overlap between the case mix of each hospital and that of all other Massachusetts hospitals to justify comparison of individual hospital performance with that of the remaining hospitals. As expected, some pairwise hospital comparisons indicated lack of comparability. This finding illustrates the fallacy of assuming that risk adjustment per se is sufficient to permit direct side-by-side comparison of healthcare providers. In some instances, such analyses may be facilitated by the use of propensity scores to improve covariate balance between institutions and to justify such comparisons. Conclusions— Risk-adjusted outcomes, commonly the focus of public report cards, have a specific interpretation. Using indirect standardization, these outcomes reflect a provider’s performance for its specific case mix relative to the expected performance of an average provider for that same case mix. Unless study design or post hoc adjustments have resulted in reasonable overlap of case-mix distributions, such risk-adjusted outcomes should not be used to directly compare one institution with another.Keywords
This publication has 42 references indexed in Scilit:
- The design versus the analysis of observational studies for causal effects: parallels with the design of randomized trialsStatistics in Medicine, 2006
- On the use of discrete choice models for causal inferenceStatistics in Medicine, 2005
- Causal Inference Using Potential OutcomesJournal of the American Statistical Association, 2005
- Appropriate Assessment of Neighborhood Effects on Individual Health: Integrating Random and Fixed Effects in Multilevel Logistic RegressionAmerican Journal of Epidemiology, 2005
- Statistical Analysis of Performance Indicators in UK Higher EducationJournal of the Royal Statistical Society Series A: Statistics in Society, 2004
- Causal Effects in Nonexperimental Studies: Reevaluating the Evaluation of Training ProgramsJournal of the American Statistical Association, 1999
- Statistical Methods for Profiling Providers of Medical Care: Issues and ApplicationsJournal of the American Statistical Association, 1997
- Statistics and Causal InferenceJournal of the American Statistical Association, 1986
- Reducing Bias in Observational Studies Using Subclassification on the Propensity ScoreJournal of the American Statistical Association, 1984
- The central role of the propensity score in observational studies for causal effectsBiometrika, 1983