Decompositions of the mean continuous ranked probability score
Open Access
- 1 January 2024
- journal article
- research article
- Published by Institute of Mathematical Statistics in Electronic Journal of Statistics
- Vol. 18 (2) , 4992-5044
Abstract
The continuous ranked probability score (crps) is the most commonly used scoring rule in the evaluation of probabilistic forecasts for real-valued outcomes. To assess and rank forecasting methods, researchers compute the mean crps over given sets of forecast situations, based on the respective predictive distributions and outcomes. We propose a new, isotonicity-based decomposition of the mean crps into interpretable components that quantify miscalibration (MCB), discrimination ability (DSC), and uncertainty (UNC), respectively. In a detailed theoretical analysis, we compare the new approach to empirical decompositions proposed earlier, generalize to population versions, analyse their properties and relationships, and relate to a hierarchy of notions of calibration. The isotonicity-based decomposition guarantees the nonnegativity of the components and quantifies calibration in a sense that is stronger than for other types of decompositions, subject to the nondegeneracy of empirical decompositions. We illustrate the usage of the isotonicity-based decomposition and miscalibration–discrimination (MCB–DSC) plots in case studies from weather prediction and machine learning.Keywords
All Related Versions
This publication has 57 references indexed in Scilit:
- Accelerating the Pool-Adjacent-Violators Algorithm for Isotonic Distributional RegressionMethodology and Computing in Applied Probability, 2022
- Receiver operating characteristic (ROC) curves: equivalences, beta model, and minimum distance estimationMachine Learning, 2021
- Characterizing the optimal solutions to the isotonic regression problem for identifiable functionalsAnnals of the Institute of Statistical Mathematics, 2021
- Understanding changes of the continuous ranked probability score using a homogeneous Gaussian approximationQuarterly Journal of the Royal Meteorological Society, 2020
- Simplifying and generalising Murphy's Brier score decompositionQuarterly Journal of the Royal Meteorological Society, 2017
- Decomposition and graphical portrayal of the quantile scoreQuarterly Journal of the Royal Meteorological Society, 2013
- Probabilistic quantitative precipitation forecasting using Ensemble Model Output StatisticsQuarterly Journal of the Royal Meteorological Society, 2013
- A note on generalized inversesMathematical Methods of Operations Research, 2013
- A bias‐corrected decomposition of the Brier scoreQuarterly Journal of the Royal Meteorological Society, 2012
- Stochastic Orders2007