Informative Evaluation Metrics for Highly Imbalanced Big Data Classification
Authors:
- 1 December 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1419-1426
Abstract
We conduct experiments that show the Area Under the Precision Recall Curve (AUPRC) metric provides a more meaningful insight into the impact of Random Undersampling than Area Under the Receiver Operating Characteristic Curve (AUC). Evaluating experiments with multiple metrics is a robust method for overcoming challenges in Machine Learning, such as class imbalance. Random Undersampling is a technique to deal with class imbalance. We find Random Undersampling may provide an improvement to AUC scores. However, at the same time, Random Undersampling may be detrimental to AUPRC scores. AUPRC is a metric that involves precision, whereas AUC does not. In the classification of imbalanced Big Data, an increase in false positive counts has a more noticeable drop in precision scores. Therefore, in application domains where false positives are undesirable, optimizing models for AUPRC is a wise choice. Our contribution is to compare the performance of models in terms of AUPRC and AUC to show the impact of Random Undersampling on the classification of imbalanced Big Data. We compare the performance via experiments in the classification of highly imbalanced Big Data. Models are built with data in its original class ratio, and with data undersampled into 5 distinct class ratios. We report the results of 600 experiments where we apply Random Undersampling to a dataset with about 175 million instances. To the best of our knowledge we are the first to utilize Medicare Part D data which became available in 2021.Keywords
This publication has 26 references indexed in Scilit:
- Medicare Fraud Detection Using Machine Learning Methods2017
- Apache SparkCommunications of the ACM, 2016
- XGBoost2016
- A Study on the Relationships of Classifier Performance Metrics2009
- Learning from Imbalanced DataIEEE Transactions on Knowledge and Data Engineering, 2009
- Extremely randomized treesMachine Learning, 2006
- Borderline-SMOTE: A New Over-Sampling Method in Imbalanced Data Sets Learning2005
- Random ForestsMachine Learning, 2001
- Greedy function approximation: A gradient boosting machine.The Annals of Statistics, 2001
- Analysis of Variance1987