Outlier Aversion in Subjective Evaluation
Author:
- 9 May 2007
- journal article
- research article
- Published by SAGE Publications in Journal of Sports Economics
- Vol. 9 (2) , 141-159
Abstract
The quality of subjective performance evaluation is dependent on the incentive structures evaluators face. Figure skating competitions provide a unique opportunity to study subjective evaluation. Using scoring data from World Figure Skating Championships between 2001 and 2003, I test for the existence of “outlier aversion” in which subjective evaluators avoid submitting outlying judgments. I find that judges manipulate scores to achieve a targeted level of agreement with the other judges. Agreement may not be a good criterion for the validity of an evaluation system, consistent with the recent applied psychology and management literature.Keywords
This publication has 20 references indexed in Scilit:
- Favoritism Under Social PressureReview of Economics and Statistics, 2005
- Expert Opinion and Compensation: Evidence from a Musical CompetitionAmerican Economic Review, 2003
- Dynamic panel data models: a guide to micro data methods and practicePortuguese Economic Journal, 2002
- Macroeconomic forecasts and microeconomic forecastersJournal of Economic Behavior & Organization, 2002
- Nonparametric Tests of the Unbiasedness of Olympic Figure-Skating JudgmentsJournal of the Royal Statistical Society: Series D (The Statistician), 1996
- A Theory of ConformityJournal of Political Economy, 1994
- Rating SkatingJournal of the American Statistical Association, 1994
- Performance Appraisal Process Research in the 1980s: What Has It Contributed to Appraisals in Use?Organizational Behavior and Human Decision Processes, 1993
- Some Tests of Specification for Panel Data: Monte Carlo Evidence and an Application to Employment EquationsReview of Economic Studies, 1991
- Validity of behavioral assessment for predicting military recruiter performance.Journal of Applied Psychology, 1982