Frugal Gaussian clustering of huge imbalanced datasets through a bin-marginal approach
Preprint
Abstract
Clustering conceptually reveals all its interest when the dataset size considerably increases since there is the opportunity to discover tiny but possibly high value clusters which were out of reach with more modest sample sizes. However, clustering is practically faced to computer limits with such high data volume, since possibly requiring extremely high memory and computation resources. In addition, the classical subsampling strategy, often adopted to overcome these limitations, is expected to heavily failed for discovering clusters in the highly imbalanced cluster case. Our proposal first consists in drastically compressing the data volume by just preserving its bin-marginal values, thus discarding the bin-cross ones. Despite this extreme information loss, we then prove identifiability property for the diagonal mixture model and also introduce a specific EM-like algorithm associated to a composite likelihood approach. This latter is extremely more frugal than a regular but unfeasible EM algorithm expected to be used on our bin-marginal data, while preserving all consistency properties. Finally, numerical experiments highlight that this proposed method outperforms subsampling both in controlled simulations and in various real applications where imbalanced clusters may typically appear, such as image segmentation, hazardous asteroids recognition and fraud detection.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Composite likelihood methods for histogram-valued random variablesStatistics and Computing, 2020
- An insight into imbalanced Big Data classification: outcomes and challengesComplex & Intelligent Systems, 2017
- A comparison of programming languages in macroeconomicsJournal of Economic Dynamics and Control, 2015
- Mixture models for ordinal data: a pairwise likelihood approachStatistics and Computing, 2014
- Learned lessons in credit card fraud detection from a practitioner perspectiveExpert Systems with Applications, 2014
- Electric sail missions to potentially hazardous asteroidsActa Astronautica, 2010
- Maximum Likelihood Estimation of Mixture Densities for Binned and Truncated Multivariate DataMachine Learning, 2002
- Adaptive Fraud DetectionData Mining and Knowledge Discovery, 1997
- Gaussian parsimonious clustering modelsPattern Recognition, 1995
- Comparing partitionsJournal of Classification, 1985