Robust and sparse k-means clustering for high-dimensional data
Authors:
Preprint
Abstract
In real-world application scenarios, the identification of groups poses a significant challenge due to possibly occurring outliers and existing noise variables. Therefore, there is a need for a clustering method which is capable of revealing the group structure in data containing both outliers and noise variables without any pre-knowledge. In this paper, we propose a $k$-means-based algorithm incorporating a weighting function which leads to an automatic weight assignment for each observation. In order to cope with noise variables, a lasso-type penalty is used in an objective function adjusted by observation weights. We finally introduce a framework for selecting both the number of clusters and variables based on a modified gap statistic. The conducted experiments on simulated and real-world data demonstrate the advantage of the method to identify groups, outliers, and informative variables simultaneously.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Modelling the role of variables in model-based cluster analysisStatistics and Computing, 2017
- Hierarchical Density Estimates for Data Clustering, Visualization, and Outlier DetectionACM Transactions on Knowledge Discovery From Data, 2015
- Snipping for robust k-means clustering under component-wise contaminationStatistics and Computing, 2013
- A comparative study of efficient initialization methods for the k-means clustering algorithmExpert Systems with Applications, 2013
- Exploring the number of groups in robust model-based clusteringStatistics and Computing, 2010
- A review of robust clustering methodsAdvances in Data Analysis and Classification, 2010
- Data clustering: 50 years beyond K-meansPattern Recognition Letters, 2010
- Outlier identification in high dimensionsComputational Statistics & Data Analysis, 2008
- Partial robust M-regressionChemometrics and Intelligent Laboratory Systems, 2005
- Estimating the Number of Clusters in a Data Set Via the Gap StatisticJournal of the Royal Statistical Society Series B: Statistical Methodology, 2001