Quantifying Randomness Versus Consensus in Wine Quality Ratings
Author:
- 29 April 2014
- journal article
- research article
- Published by Cambridge University Press (CUP) in Journal of Wine Economics
- Vol. 9 (2) , 202-213
Abstract
There has been ongoing interest in studying wine judges' performance in evaluating wines. Most of the studies have reached a similar conclusion: a significant lack of consensus exists in wine quality ratings. However, a few studies, to the author's knowledge, have provided direct quantification of how much consensus (as opposed to randomness) exists in wine ratings. In this paper, a permutation-based mixed model is proposed to quantify randomness versus consensus in wine ratings. Specifically, wine ratings under the condition of randomness are generated with a permutation method, and wine ratings under the condition of consensus can be produced by sorting the ratings for each judge. Then the observed wine ratings are modeled as a mixture of ratings under randomness and ratings under consensus. This study shows that the model can provide excellent model fit, which indicates that wine ratings, indeed, consist of a mixture of randomness and consensus. A direct measure is easily computed to quantify randomness versus consensus in wine ratings. The method is demonstrated with data analysis from a major wine competition and a simulation study. (JEL Classifications: C10, C13, C15)Keywords
This publication has 11 references indexed in Scilit:
- Reliability and Consensus of Experienced Wine Judges: Expertise Within and Between?Journal of Wine Economics, 2012
- Evaluation of Wine Judge Performance through Three Characteristics: Bias, Discrimination, and VariationJournal of Wine Economics, 2010
- An Analysis of the Concordance Among 13 U.S. Wine CompetitionsJournal of Wine Economics, 2009
- Evaluation of the consistency of wine quality assessments from expert wine tastersAustralian Journal of Grape and Wine Research, 2008
- An Examination of Judge Reliability at a major U.S. Wine CompetitionJournal of Wine Economics, 2008
- Improved statistical tests for differential gene expression by shrinking variance components estimatesBiostatistics, 2004
- Significance analysis of microarrays applied to the ionizing radiation responseProceedings of the National Academy of Sciences, 2001
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960
- III. Contributions to the mathematical theory of evolutionPhilosophical Transactions of the Royal Society of London. (A.), 1894