Precision and Reliability of Forecasts Performance Metrics
Open Access
- 30 October 2022
- journal article
- research article
- Published by MDPI AG in Forecasting
- Vol. 4 (4) , 882-903
Abstract
The selection of an accurate performance metric is highly important to evaluate the quality of a forecasting method. This evaluation may help to select between different forecasting tools of forecasting outputs, and then support many decisions within a company. This paper proposes to evaluate the sensitivity and reliability of forecasts performance metrics. The methodology is tested using multiple time series of different scales and demand patterns, such as intermittent demand. The idea is to add to each series a noise following a known distribution to represent forecasting models of a known error distribution. Varying the parameters of the distribution of the noise allows to evaluate how sensitive and reliable performance metrics are to changes in bias and variance of the error of a forecasting model. The experiments concluded that sRMSE is more reliable than MASE in most cases on those series. sRMSE is especially reliable for detecting changes in the variance of a model and sPIS is the most sensitive metric to the bias of a model. sAPIS is sensible to both variance and bias but is less reliable.Keywords
Funding Information
- Mitacs (IT 12058)
This publication has 16 references indexed in Scilit:
- Evaluation of forecasting error measurements and techniques for intermittent demandInternational Journal of Production Economics, 2010
- Forecasting intermittent demand: a comparative studyJournal of the Operational Research Society, 2009
- Another look at measures of forecast accuracyInternational Journal of Forecasting, 2006
- Exponential smoothing model selection for forecastingInternational Journal of Forecasting, 2006
- The accuracy of intermittent demand estimatesInternational Journal of Forecasting, 2005
- Cumulated gain-based evaluation of IR techniquesACM Transactions on Information Systems, 2002
- The M3-Competition: results, conclusions and implicationsInternational Journal of Forecasting, 2000
- On the asymmetry of the symmetric MAPEInternational Journal of Forecasting, 1999
- The Proof and Measurement of Association between Two ThingsThe American Journal of Psychology, 1987
- Forecasting and Stock Control for Intermittent DemandsJournal of the Operational Research Society, 1972