Assessing the calibration of multivariate probabilistic forecasts
Authors:
- 4 February 2024
- journal article
- research article
- Published by Wiley in Quarterly Journal of the Royal Meteorological Society
- Vol. 150 (760) , 1315-1335
Abstract
Rank and probability integral transform histograms are established tools to assess the calibration of probabilistic forecasts. They not only check whether a forecast is calibrated, but they also reveal what systematic biases (if any) are present in the forecasts. Several extensions of rank histograms have been proposed to evaluate the calibration of probabilistic forecasts for multivariate outcomes. These extensions introduce a so‐called pre‐rank function that condenses the multivariate forecasts and observations into univariate objects, from which a standard rank histogram can be produced. Existing pre‐rank functions typically aim to preserve as much information as possible when condensing the multivariate forecasts and observations into univariate objects. Although this is sensible when conducting statistical tests for multivariate calibration, it can hinder the interpretation of the resulting histograms. In this article, we demonstrate that there are few restrictions on the choice of pre‐rank function, meaning forecasters can choose a pre‐rank function depending on what information they want to extract concerning forecast performance. We introduce the concept of simple pre‐rank functions and provide examples that can be used to assess the mean, spread, and dependence structure of multivariate probabilistic forecasts, as well as pre‐rank functions that could be useful when evaluating probabilistic spatial field forecasts. The simple pre‐rank functions that we introduce are easy to interpret, easy to implement, and they deliberately provide complementary information, meaning several pre‐rank functions can be employed to achieve a more complete understanding of multivariate forecast performance. We then discuss how e‐values can be employed to formally test for multivariate calibration over time. This is demonstrated in an application to wind‐speed forecasting using the EUPPBench post‐processing benchmark dataset.Keywords
All Related Versions
This publication has 44 references indexed in Scilit:
- Verifying Forecasts SpatiallyBulletin of the American Meteorological Society, 2010
- Verification with VariogramsWeather and Forecasting, 2009
- SAL—A Novel Quality Measure for the Verification of Quantitative Precipitation ForecastsMonthly Weather Review, 2008
- Scale-Selective Verification of Rainfall Accumulations from High-Resolution Forecasts of Convective EventsMonthly Weather Review, 2008
- Probabilistic Forecasts, Calibration and SharpnessJournal of the Royal Statistical Society Series B: Statistical Methodology, 2007
- Strictly Proper Scoring Rules, Prediction, and EstimationJournal of the American Statistical Association, 2007
- Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS EstimationMonthly Weather Review, 2005
- The Minimum Spanning Tree Histogram as a Verification Tool for Multidimensional Ensemble ForecastsMonthly Weather Review, 2004
- Evaluating Density Forecasts with Applications to Financial Risk ManagementInternational Economic Review, 1998
- The Contiguity Ratio and Statistical MappingThe Incorporated Statistician, 1954