The case for quantile forecasts: Enhancing reliability in the identification of forecaster talent
Open Access
- 14 October 2026
- journal article
- research article
- Published by Cambridge University Press (CUP) in Judgment and Decision Making
- Vol. 21, e29
Abstract
What is the best way to elicit forecasts if the goal is to measure the skill of forecasters? For continuous outcomes, there are two common approaches: elicit probabilities at fixed quantiles of the outcome distribution (e.g., what is the probability gas will cost less than $4.00 per gallon in 1 month?), or elicit quantiles at fixed probabilities (e.g., what is the price such that there is a 25% chance a gallon of gas will cost less than it in 1 month?). We compare these methods using a simulation study and a longitudinal survey in which 1,194 participants provided five waves of forecasts in both formats on a total of 36 real-world questions. Although probability elicitation is more ubiquitous in the literature, we show in our simulation and survey data that quantile elicitation allows us to estimate forecaster skill more efficiently with less measurement error. While quantile elicitation is time-consuming and associated with comprehension difficulty indicated by incoherent forecasts (violations of monotonicity), there is a clear reduction in incoherence across survey waves, suggesting comprehension of the quantile elicitation format improves with practice and is trainable.Keywords
This publication has 46 references indexed in Scilit:
- Identifying good forecasters via adaptive cognitive testsInternational Journal of Forecasting, 2026
- Measuring probabilistic coherence to identify superior forecastersInternational Journal of Forecasting, 2024
- Talent Spotting in Crowd Prediction2023
- Performance-Weighted Aggregation: Ferreting Out Wisdom Within the Crowd2023
- Hybrid forecasting of geopolitical events†AI Magazine, 2023
- History and nature of the Jeffreys–Lindley paradoxArchive for History of Exact Sciences, 2022
- Preference for human or algorithmic forecasting advice does not predict if and how it is usedJournal of Behavioral Decision Making, 2022
- Default Bayes factors for ANOVA designsJournal of Mathematical Psychology, 2012
- Measuring Risk Literacy: The Berlin Numeracy TestJudgment and Decision Making, 2012
- Coefficient Alpha and the Internal Structure of TestsPsychometrika, 1951