Fragility Index: A New Approach for Binary Classification
Open Access
- 4 August 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2918-2929
Abstract
In binary classification problems, many performance metrics evaluate the probability that some error exceeds a threshold. Nevertheless, they focus more on the probability and fail to capture the magnitude of the error, which evaluates how large this error exceeds the threshold. Capturing the magnitude of error is desired in many applications. For example, in detecting disease and predicting credit default, the magnitude of error illustrates the confidence in making the wrong prediction. We propose a novel metric, the Fragility Index (FI), to evaluate the performance of binary classifiers by capturing the magnitude of the error. FI alleviates the risk of misclassification by penalizing the large error greatly, which is seldom considered by standard metrics. Moreover, to strengthen the generalization ability and handle unseen samples, we adopt the framework of distributionally robust optimization and robust satisficing, which allows us to derive and control the maximum degree of fragility of the classifier when the distribution of samples shifts. We show that FI can be easily calculated and optimized for common probabilistic distance measures. Experiments with real datasets demonstrate the new insights brought by FI and the advantages of classifiers selected under FI, which always improve the robustness and reduce the risk of large errors as compared to classifiers selected by alternative metrics.Keywords
Funding Information
- Hong Kong Research Grants Council (Grants 16204521 16212419)
- Alibaba Innovative Research Program
This publication has 22 references indexed in Scilit:
- Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulationsMathematical Programming, 2017
- Credit default prediction modeling: an application of support vector machineRisk Management, 2017
- Data-driven chance constrained stochastic programMathematical Programming, 2015
- Robust Solutions of Optimization Problems Affected by Uncertain ProbabilitiesManagement Science, 2013
- Nonlinear regression model generation using hyperparameter optimizationComputers & Mathematics with Applications, 2010
- Physician Cost Profiling — Reliability and Risk of MisclassificationNew England Journal of Medicine, 2010
- Machine Learning for Detection and Diagnosis of DiseaseAnnual Review of Biomedical Engineering, 2006
- Use of some sensitivity criteria for choosing networks with good generalization abilityNeural Processing Letters, 1995
- The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology, 1982
- Comparison of the predicted and observed secondary structure of T4 phage lysozymeBiochimica et Biophysica Acta (BBA) - Protein Structure, 1975