Attila Fazekas

A central challenge in machine learning research is to develop robust classification models from imbalanced datasets. One possible solution is to generate synthetic samples to augment the underrepresented classes. Although the effectiveness of oversampling has been demonstrated in numerous studies, it has also been criticized for propagating label noise within the dataset. Sampling methods employ various strategies to mitigate this issue, but our knowledge of their actual effectiveness is limited. In this study, we introduce an ancestor‐based measure, termed impurity, to quantify the impact of oversamplers on class‐label noise. We compare 60 techniques designed for low‐dimensional, small‐imbalance datasets, across both synthetic and real‐world data. In addition to the impurity level, we also examine the preservation rate of correctly labelled samples and the degree of class overlap to obtain a comprehensive picture of the effects of oversampling methods. Several methods lowered impurity while still preserving valid minority samples, whereas others propagated noise more strongly than standard SMOTE. Approaches combining moderate noise filtering with supervised sample generation provided the most balanced trade‐off. Analysis of the training sets associated with the best‐performing models revealed that higher predictive performance was generally accompanied by lower impurity, while changes in class overlap were less systematic. Oversamplers that introduced substantial label noise or failed to preserve correctly labelled samples were less likely to be associated with the best-performing models selected according to cross-validated F1 scores. Overall, these findings indicate that, within the scope of our study, cross‐validation can offer meaningful guidance when selecting an oversampling strategy.

A very interesting paper that deals with the fact that quality scores used to characterize generative models often give better results in two-dimensional cases than they should. This is the so-called "grade inflation problem". The limitation of the a...

K-fold cross-validation is a widely used tool for assessing classifier performance. The reproducibility crisis faced by artificial intelligence partly results from the irreproducibility of reported k-fold cross-validation-based performance scores. Recently, we introduced numerical techniques to test the consistency of claimed performance scores and experimental setups. In a crucial use case, the method relies on the combinatorial enumeration of all k-fold configurations, for which we proposed an algorithm in the binary classification case.

Addressing the reproducibility crisis in artificial intelligence through the validation of reported experimental results is a challenging task. It necessitates either the reimplementation of techniques or a meticulous assessment of papers for deviations from the scientific method and best statistical practices. To facilitate the validation of reported results, we have developed numerical techniques capable of identifying inconsistencies between reported performance scores and various experimental setups in machine learning problems, including binary/multiclass classification and regression. These consistency tests are integrated into the open-source package mlscorecheck, which also provides specific test bundles designed to detect systematically recurring flaws in various fields, such as retina image processing and synthetic minority oversampling.

Binary classification is a fundamental task in machine learning, with applications spanning various scientific domains. Whether scientists are conducting fundamental research or refining practical applications, they typically assess and rank classification techniques based on performance metrics such as accuracy, sensitivity, and specificity. However, reported performance scores may not always serve as a reliable basis for research ranking. This can be attributed to undisclosed or unconventional practices related to cross-validation, typographical errors, and other factors. In a given experimental setup, with a specific number of positive and negative test items, most performance scores can assume specific, interrelated values. In this paper, we introduce numerical techniques to assess the consistency of reported performance scores and the assumed experimental setup. Importantly, the proposed approach does not rely on statistical inference but uses numerical methods to identify inconsistencies with certainty. Through three different applications related to medicine, we demonstrate how the proposed techniques can effectively detect inconsistencies, thereby safeguarding the integrity of research fields. To benefit the scientific community, we have made the consistency tests available in an open-source Python package.

Nowadays, the solution to many practical problems relies on machine learning tools. However, compiling the appropriate training data set for real-world classification problems is challenging because collecting the right amount of data for each class is often difficult or even impossible. In such cases, we can easily face the problem of imbalanced learning. There are many methods in the literature for solving the imbalanced learning problem, so it has become a serious question how to compare the performance of the imbalanced learning methods. Inadequate validation techniques can provide misleading results (e.g., due to data shift), which leads to the development of methods designed for imbalanced data sets, such as stratified cross-validation (SCV) and distribution optimally balanced SCV (DOB-SCV). Previous studies have shown that higher classification performance scores (AUC) can be achieved on imbalanced data sets using DOB-SCV instead of SCV. We investigated the effect of the oversamplers on this difference. The study was conducted on 420 data sets, involving several sampling methods and the DTree, kNN, SVM, and MLP classifiers. We point out that DOB-SCV often provides a little higher F1 and AUC values for classification combined with sampling. However, the results also prove that the selection of the sampler–classifier pair is more important for the classification performance than the choice between the DOB-SCV and the SCV techniques.