Loss function based ranking in two-stage, hierarchical models
Open Access
- 1 December 2006
- journal article
- Published by Institute of Mathematical Statistics in Bayesian Analysis
- Vol. 1 (4) , 915-946
Abstract
Performance evaluations of health services providers burgeons. Similarly, analyzing spatially related health information, ranking teachers and schools, and identification of differentially expressed genes are increasing in prevalence and importance. Goals include valid and efficient ranking of units for profiling and league tables, identification of excellent and poor performers, the most differentially expressed genes, and determining "exceedances" (how many and which unit-specific true parameters exceed a threshold). These data and inferential goals require a hierarchical, Bayesian model that accounts for nesting relations and identifies both population values and random effects for unit-specific parameters. Furthermore, the Bayesian approach coupled with optimizing a loss function provides a framework for computing non-standard inferences such as ranks and histograms. Estimated ranks that minimize Squared Error Loss (SEL) between the true and estimated ranks have been investigated. The posterior mean ranks minimize SEL and are "general purpose," relevant to a broad spectrum of ranking goals. However, other loss functions and optimizing ranks that are tuned to application-specific goals require identification and evaluation. For example, when the goal is to identify the relatively good (e.g., in the upper 10%) or relatively poor performers, a loss function that penalizes classification errors produces estimates that minimize the error rate. We construct loss functions that address this and other goals, developing a unified framework that facilitates generating candidate estimates, comparing approaches and producing data analytic performance summaries. We compare performance for a fully parametric, hierarchical model with Gaussian sampling distribution under Gaussian and a mixture of Gaussians prior distributions. We illustrate approaches via analysis of standardized mortality ratio data from the United States Renal Data System. Results show that SEL-optimal ranks perform well over a broad class of loss functions but can be improved upon when classifying units above or below a percentile cut-point. Importantly, even optimal rank estimates can perform poorly in many real-world settings; therefore, data-analytic performance summaries should always be reported.Keywords
This publication has 32 references indexed in Scilit:
- Statistical Analysis of Performance Indicators in UK Higher EducationJournal of the Royal Statistical Society Series A: Statistics in Society, 2004
- Loss functions for estimation of extrema with an application to disease mappingCanadian Journal of Statistics, 2003
- Selection of Related Multivariate MeansJournal of the American Statistical Association, 2003
- A Direct Approach to False Discovery RatesJournal of the Royal Statistical Society Series B: Statistical Methodology, 2002
- Significance analysis of microarrays applied to the ionizing radiation responseProceedings of the National Academy of Sciences, 2001
- Bayesian Data Mining in Large Frequency Tables, with an Application to the FDA Spontaneous Reporting SystemThe American Statistician, 1999
- Triple-goal Estimates in Two-stage Hierarchical ModelsJournal of the Royal Statistical Society Series B: Statistical Methodology, 1998
- Statistical Methods for Profiling Providers of Medical Care: Issues and ApplicationsJournal of the American Statistical Association, 1997
- Empirical bayes estimators for spatially correlated incidence ratesEnvironmetrics, 1994
- A constrained empirical bayes estimator for incidence rates in areas with small populationsStatistics in Medicine, 1994