Evaluation of a Performance-Based Expert Elicitation: WHO Global Attribution of Foodborne Diseases
Open Access
- 1 March 2016
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 11 (3) , e0149817
Abstract
For many societally important science-based decisions, data are inadequate, unreliable or non-existent, and expert advice is sought. In such cases, procedures for eliciting structured expert judgments (SEJ) are increasingly used. This raises questions regarding validity and reproducibility. This paper presents new findings from a large-scale international SEJ study intended to estimate the global burden of foodborne disease on behalf of WHO. The study involved 72 experts distributed over 134 expert panels, with panels comprising thirteen experts on average. Elicitations were conducted in five languages. Performance-based weighted solutions for target questions of interest were formed for each panel. These weights were based on individual expert’s statistical accuracy and informativeness, determined using between ten and fifteen calibration variables from the experts' field with known values. Equal weights combinations were also calculated. The main conclusions on expert performance are: (1) SEJ does provide a science-based method for attribution of the global burden of foodborne diseases; (2) equal weighting of experts per panel increased statistical accuracy to acceptable levels, but at the cost of informativeness; (3) performance-based weighting increased informativeness, while retaining accuracy; (4) due to study constraints individual experts’ accuracies were generally lower than in other SEJ studies, and (5) there was a negative correlation between experts' informativeness and statistical accuracy which attenuated as accuracy improved, revealing that the least accurate experts drive the negative correlation. It is shown, however, that performance-based weighting has the ability to yield statistically accurate and informative combinations of experts' judgments, thereby offsetting this contrary influence. The present findings suggest that application of SEJ on a large scale is feasible, and motivate the development of enhanced training and tools for remote elicitation of multiple, internationally-dispersed panels.Keywords
This publication has 11 references indexed in Scilit:
- Research Synthesis Methods in an Age of Globalized Risks: Lessons from the Global Burden of Foodborne Disease Expert ElicitationRisk Analysis, 2016
- World Health Organization Estimates of the Relative Contributions of Food to the Burden of Disease Due to Selected Foodborne Hazards: A Structured Expert ElicitationPLOS ONE, 2016
- World Health Organization Global Estimates and Regional Comparisons of the Burden of Foodborne Disease in 2010PLOS Medicine, 2015
- Messaging climate change uncertaintyNature Climate Change, 2014
- Out-of-sample validation for structured expert judgment of Asian carp establishment in Lake ErieIntegrated Environmental Assessment and Management, 2014
- The effect of the number of seed variables on the performance of Cooke′s classical modelReliability Engineering & System Safety, 2014
- A route to more tractable expert adviceNature, 2010
- Attributing the Human Disease Burden of Foodborne Infections to Specific SourcesFoodborne Pathogens and Disease, 2009
- Uncertain Judgements: Eliciting Experts' Probabilities2006
- Combining Probability Distributions From Experts in Risk AnalysisRisk Analysis, 1999