Escaping the curse of dimensionality in Bayesian model based clustering
Preprint
Abstract
Bayesian mixture models are widely used for clustering of high-dimensional data with appropriate uncertainty quantification. However, as the dimension of the observations increases, posterior inference often tends to favor too many or too few clusters. This article explains this behavior by studying the random partition posterior in a non-standard setting with a fixed sample size and increasing data dimensionality. We provide conditions under which the finite sample posterior tends to either assign every observation to a different cluster or all observations to the same cluster as the dimension grows. Interestingly, the conditions do not depend on the choice of clustering prior, as long as all possible partitions of observations into clusters have positive prior probabilities, and hold irrespective of the true data-generating model. We then propose a class of latent mixtures for Bayesian clustering (Lamb) on a set of low-dimensional latent variables inducing a partition on the observed data. The model is amenable to scalable posterior inference and we show that it can avoid the pitfalls of high-dimensionality under mild assumptions. The proposed approach is shown to have good performance in simulation studies and an application to inferring cell types based on scRNAseq.Keywords
All Related Versions
This publication has 43 references indexed in Scilit:
- Integrating single-cell transcriptomic data across different conditions, technologies, and speciesNature Biotechnology, 2018
- Reference component analysis of single-cell transcriptomes elucidates cellular heterogeneity in human colorectal tumorsNature Genetics, 2017
- Model-based clustering of high-dimensional data: A reviewComputational Statistics & Data Analysis, 2014
- Accounting for technical noise in single-cell RNA-seq experimentsNature Methods, 2013
- Using conditional independence for parsimonious model-based Gaussian clusteringStatistics and Computing, 2012
- Introduction to the non-asymptotic analysis of random matrices2012
- Penalized factor mixture analysis for variable selection in clustered dataComputational Statistics & Data Analysis, 2009
- High dimensional covariance matrix estimation using a factor modelJournal of Econometrics, 2008
- Finite Mixture and Markov Switching Models2006
- Silhouettes: A graphical aid to the interpretation and validation of cluster analysisJournal of Computational and Applied Mathematics, 1987