SMOTE-BD: An Exact and Scalable Oversampling Method for Imbalanced Classification in Big Data
Authors:
- María José BasgallMaría José Basgall
- Waldo HasperuéWHWaldo Hasperué
- Marcelo NaioufMNMarcelo Naiouf
Open Access
- 12 December 2018
- journal article
- research article
- Published by Universidad Nacional de La Plata in Journal of Computer Science and Technology
- Vol. 18 (03) , e23
Abstract
The volume of data in today's applications has meant a change in the way Machine Learning issues are addressed. Indeed, the Big Data scenario involves scalability constraints that can only be achieved through intelligent model design and the use of distributed technologies. In this context, solutions based on the Spark platform have established themselves as a de facto standard. In this contribution, we focus on a very important framework within Big Data Analytics, namely classification with imbalanced datasets. The main characteristic of this problem is that one of the classes is underrepresented, and therefore it is usually more complex to find a model that identifies it correctly. For this reason, it is common to apply preprocessing techniques such as oversampling to balance the distribution of examples in classes. In this work we present SMOTE-BD, a fully scalable preprocessing approach for imbalanced classification in Big Data. It is based on one of the most widespread preprocessing solutions for imbalanced classification, namely the SMOTE algorithm, which creates new synthetic instances according to the neighborhood of each example of the minority class. Our novel development is made to be independent of the number of partitions or processes created to achieve a higher degree of efficiency. Experiments conducted on different standard and Big Data datasets show the quality of the proposed design and implementation.Keywords
This publication has 9 references indexed in Scilit:
- Big Data: Tutorial and guidelines on information and process fusion for analytics algorithms with MapReduceInformation Fusion, 2018
- Surveying alignment-free features for Ortholog detection in related yeast proteomes by using supervised big data classifiersBMC Bioinformatics, 2018
- SMOTE for Learning from Imbalanced Data: Progress and Challenges, Marking the 15-year AnniversaryJournal of Artificial Intelligence Research, 2018
- An insight into imbalanced Big Data classification: outcomes and challengesComplex & Intelligent Systems, 2017
- kNN-IS: An Iterative Spark-based design of the k-Nearest Neighbors classifier for big dataKnowledge-Based Systems, 2017
- Learning from imbalanced data: open challenges and future directionsProgress in Artificial Intelligence, 2016
- Class imbalance revisited: a new experimental setup to assess the performance of treatment methodsKnowledge and Information Systems, 2014
- Data-intensive applications, challenges, techniques and technologies: A survey on Big DataInformation Sciences, 2014
- SMOTE: Synthetic Minority Over-sampling TechniqueJournal of Artificial Intelligence Research, 2002