On the Evaluation of Effort Estimation Models
Open Access
- 15 June 2017
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 41-50
Abstract
Background. Using accurate effort estimation models can help software companies plan, monitor, and control their development process and development costs. It is therefore important to define sound accuracy indicators that allow practitioners and researchers to assess and rank different effort estimation models so that practitioners can select the most accurate, and therefore useful one. Several accuracy indicators exist, with different advantages and disadvantages. Objective. We propose a general framework for building sound accuracy indicators for effort estimation models. Method. The accuracy indicators that comply with our proposal are built by means of a comparison between a reference effort estimation model and the specific model whose accuracy we would like to assess. Several existing indicators are built this way: we develop a framework so new indicators can be defined in a sound way. Results. From a theoretical point of view, we applied our approach to accuracy indicators based on the square of the residuals and the absolute value of the residuals. We show that using a random model as a reference model, as done in some recent literature, sets too low a bar in terms of what may be acceptable. Instead, we use reference models that are built based on constant functions. From a practical point of view, we applied our approach to datasets containing measures of industrial software development projects. With the proposed method we were able to derive indications both according to criteria already proposed in the literature and according to new criteria. Conclusions. Our method can be used to define sound accuracy indicators for effort estimation modelsKeywords
This publication has 13 references indexed in Scilit:
- Exact Mean Absolute Error of Baseline Predictor, MARP0Information and Software Technology, 2016
- Validity and reliability of evaluation procedures in comparative studies of effort prediction modelsEmpirical Software Engineering, 2011
- Building statistically significant robust regression models in empirical software engineering2009
- A Replication of the Use of Regression towards the Mean (R2M) as an Adjustment to Effort Estimation Models2005
- Reliability and validity in comparative studies of software prediction modelsIEEE Transactions on Software Engineering, 2005
- A simulation study of the model evaluation criterion mmreIEEE Transactions on Software Engineering, 2003
- What accuracy statistics really measureIEE Proceedings - Software, 2001
- Experience with the accuracy of software maintenance task effort prediction modelsIEEE Transactions on Software Engineering, 1995
- Robust regression for developing software estimation modelsJournal of Systems and Software, 1994
- Probability Distributions and Characteristic Functions1975