Why Are Experts Correlated? Decomposing Correlations Between Judges
- 1 September 2009
- journal article
- theory and-methods
- Published by Cambridge University Press (CUP) in Psychometrika
- Vol. 74 (3) , 531-553
Abstract
We derive an analytic model of the inter-judge correlation as a function of five underlying parameters. Inter-cue correlation and the number of cues capture our assumptions about the environment, while differentiations between cues, the weights attached to the cues, and (un)reliability describe assumptions about the judges. We study the relative importance of, and interrelations between these five factors with respect to inter-judge correlation. Results highlight the centrality of the inter-cue correlation. We test the model’s predictions with empirical data and illustrate its relevance. For example, we show that, typically, additional judges increase efficacy at a greater rate than additional cues.Keywords
This publication has 22 references indexed in Scilit:
- Aggregation of opinions based on correlated cues and advisorsJournal of Behavioral Decision Making, 2006
- The dominance analysis approach for comparing predictors in multiple regression.Psychological Methods, 2003
- Combining the judgments of experts: How many and which ones?Organizational Behavior and Human Decision Processes, 1986
- Aggregating Subjective Forecasts: Some Empirical ResultsManagement Science, 1985
- Limits for the Precision and Value of Information from Dependent SourcesOperations Research, 1985
- The robust beauty of improper linear models in decision making.American Psychologist, 1979
- A note on aggregating opinionsOrganizational Behavior and Human Performance, 1978
- Expert judgment: Some necessary conditions and an example.Journal of Applied Psychology, 1974
- A research paradigm for the study of interpersonal learning.Psychological Bulletin, 1966
- Some methodological considerations in multiple-cue probability studies.Psychological Review, 1964