Optimization Algorithms for Scalable Stream Batch Clustering with k Estimation
Open Access
- 25 June 2022
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 12 (13) , 6464
Abstract
The increasing volume and velocity of the continuously generated data (data stream) challenge machine learning algorithms, which must evolve to fit real-world problems. The data stream clustering algorithms face issues such as the rapidly increasing volume of the data, the variety of the number of clusters, and their shapes. The present work aims to improve the accuracy of sequential clustering batches of data streams for scenarios in which clusters evolve dynamically and continuously, automatically estimating their number. In order to achieve this goal, three evolutionary algorithms are presented, along with three novel algorithms designed to deal with clusters of normal distribution based on goodness-of-fit tests in the context of scalable batch stream clustering with automatic estimation of the number of clusters. All of them are developed on top of MapReduce, Discretized-Stream models, and the most recent MPC frameworks to provide scalability, reliability, resilience, and flexibility. The proposed algorithms are experimentally compared with state-of-the-art methods and present the best results for accuracy for normally distributed data sets, reaching their goal.Keywords
Funding Information
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior—Brasil (001, 2019/09817-6)
- Fundação de Amparo à Pesquisa do Estado de São Paulo—FAPESP (001, 2019/09817-6)
- CNPq (001, 2019/09817-6)
- FAPEMIG (001, 2019/09817-6)
This publication has 20 references indexed in Scilit:
- Improving k-means through distributed scalable metaheuristicsNeurocomputing, 2017
- An evolutionary algorithm for clustering data streams with a variable number of clustersExpert Systems with Applications, 2017
- Apache SparkCommunications of the ACM, 2016
- StreamDM: Advanced Data Mining in Spark Streaming2015
- Data stream clusteringACM Computing Surveys, 2013
- Efficiency issues of evolutionary k-meansApplied Soft Computing, 2011
- MapReduceCommunications of the ACM, 2008
- Comparing partitionsJournal of Classification, 1985
- An analysis of variance test for normality (complete samples)Biometrika, 1965
- The Kolmogorov-Smirnov Test for Goodness of FitJournal of the American Statistical Association, 1951