Discriminatory Power - An Obsolete Validation Criterion?
Authors:
- 1 February 2008
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
In this paper we analyse two common measures of discriminatory power - the Accuracy Ratio and the Area Under the Receiver Operator Characteristic - in a probabilistic framework. Under the assumption of a random default event, we verify that the measures will be portfolio dependent as discovered by Hamerle et al. (2003) and furthermore stochastic as indicated by Blochwitz et al. (2005). As an extension of these two papers we first study how the structure of a portfolio influences the measures. Furthermore we demonstrate that the measures contain information about a rating system's calibration quality. To this end a testing procedure is developed that also allows for a comparison of the measures across different portfolios. Our analysis leads to the final conclusion that high granularity and good calibration quality are sufficient to maximize attainable discriminatory power and are thus the sole criteria to be considered in any validation exercise.Keywords
This publication has 16 references indexed in Scilit:
- Economic benefit of powerful credit scoringJournal of Banking & Finance, 2006
- Myth and reality of discriminatory power for rating systemsWilmott, 2005
- Categorical Data Analysis2002
- Generally accepted rating principles: A primerJournal of Banking & Finance, 2001
- A survey of business failures with an emphasis on prediction methods and industrial applicationsEuropean Journal of Operational Research, 1996
- Goodness‐of‐Fit Testing for the Logistic Regression Model when the Estimated Probabilities are SmallBiometrical Journal, 1988
- Probabilistic prediction in patient management and clinical trialsStatistics in Medicine, 1986
- The success of business failure prediction modelsJournal of Banking & Finance, 1984
- Goodness of fit tests for the multiple logistic regression modelCommunications in Statistics - Theory and Methods, 1980
- The area above the ordinal dominance graph and the area below the receiver operating characteristic graphJournal of Mathematical Psychology, 1975