A Generic Forecast Verification Framework for Administrative Purposes
Authors:
- 1 January 2009
- journal article
- Published by American Meteorological Society in Monthly Weather Review
- Vol. 137 (1) , 331-349
Abstract
There are numerous reasons for calculating forecast verification scores, and considerable attention has been given to designing and analyzing the properties of scores that can be used for scientific purposes. Much less attention has been given to scores that may be useful for administrative reasons, such as communicating changes in forecast quality to bureaucrats and providing indications of forecast quality to the general public. The two-alternative forced choice (2AFC) test is proposed as a scoring procedure that is sufficiently generic to be usable on forecasts ranging from simple yes–no forecasts of dichotomous outcomes to forecasts of continuous variables, and can be used with deterministic or probabilistic forecasts without seriously reducing the more complex information when available. Although, as with any single verification score, the proposed test has limitations, it does have broad intuitive appeal in that the expected score of an unskilled set of forecasts (random guessing or perpetually identical forecasts) is 50%, and is interpretable as an indication of how often the forecasts are correct, even when the forecasts are expressed probabilistically and/or the observations are not discrete.Keywords
This publication has 21 references indexed in Scilit:
- The First Decade of Long-Lead U.S. Seasonal ForecastsBulletin of the American Meteorological Society, 2008
- Proper Scores for Probability Forecasts Can Never Be EquitableMonthly Weather Review, 2008
- Accounting for the effect of observation errors on verification of MOGREPSMeteorological Applications, 2008
- Understanding forecast verification statisticsMeteorological Applications, 2008
- How much does simplification of probability forecasts reduce forecast quality?Meteorological Applications, 2008
- Scoring Probabilistic Forecasts: The Importance of Being ProperWeather and Forecasting, 2007
- DEVELOPMENT OF A EUROPEAN MULTIMODEL ENSEMBLE SYSTEM FOR SEASONAL-TO-INTERANNUAL PREDICTION (DEMETER)Bulletin of the American Meteorological Society, 2004
- P stands for ?Weather, 2004
- Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretationQuarterly Journal of the Royal Meteorological Society, 2002
- What Is a Good Forecast? An Essay on the Nature of Goodness in Weather ForecastingWeather and Forecasting, 1993