The Effect of Balancing Methods on Model Behavior in Imbalanced Classification Problems
Preprint
Abstract
Imbalanced data poses a significant challenge in classification as model performance is affected by insufficient learning from minority classes. Balancing methods are often used to address this problem. However, such techniques can lead to problems such as overfitting or loss of information. This study addresses a more challenging aspect of balancing methods - their impact on model behavior. To capture these changes, Explainable Artificial Intelligence tools are used to compare models trained on datasets before and after balancing. In addition to the variable importance method, this study uses the partial dependence profile and accumulated local effects techniques. Real and simulated datasets are tested, and an open-source Python package edgaro is developed to facilitate this analysis. The results obtained show significant changes in model behavior due to balancing methods, which can lead to biased models toward a balanced distribution. These findings confirm that balancing analysis should go beyond model performance comparisons to achieve higher reliability of machine learning models. Therefore, we propose a new method performance gain plot for informed data balancing strategy to make an optimal selection of balancing method by analyzing the measure of change in model behavior versus performance gain.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- A comprehensive taxonomy for explainable artificial intelligence: a systematic survey of surveys on methods and conceptsData Mining and Knowledge Discovery, 2023
- Towards an Integrated Evaluation Framework for XAI: An Experimental StudyProcedia Computer Science, 2022
- Effect of data resampling on feature importance in imbalanced blockchain data: comparison studies of resampling techniquesData Science and Management, 2022
- A novel Random Forest integrated model for imbalanced data classification problemKnowledge-Based Systems, 2022
- A Relabeling Approach to Handling the Class Imbalance Problem for Logistic RegressionJournal of Computational and Graphical Statistics, 2021
- No Free Lunch in imbalanced learningKnowledge-Based Systems, 2021
- Automated imbalanced classification via meta-learningExpert Systems with Applications, 2021
- Explanatory Model Analysis2021
- Comparison of feature importance measures as explanations for classification modelsSN Applied Sciences, 2021
- Statistical stability indices for LIME: Obtaining reliable explanations for machine learning modelsJournal of the Operational Research Society, 2021