Efficient Density Estimation for High-Dimensional Data
Open Access
- 7 February 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 16592-16608
Abstract
Multivariate density estimation methods, typically work well in low dimensions and their extension to data analytics in high dimensions domain has proven challenging. For density estimation in high-dimensional big data domains, the non-parametric Bayesian sequential partitioning (BSP) algorithm provides an efficient way of partitioning the sample space, based on Bayesian inference. In this paper, we present a detailed analysis of BSP and provide a computationally efficient copula-transformed data structure and algorithm for use in density estimation for data analytics in high dimensions. Using the copula-transformed data structure, we implement the density estimation for marginals in both BSP and kernel density estimation (KDE) methods. The data structures and algorithm are suitably designed for most efficient rendering into parallel processing paradigms of open multi-processing (OPENMP®) and message passing interface (MPI).Keywords
This publication has 37 references indexed in Scilit:
- Optimal Bandwidth Selection for Kernel Density Functionals EstimationJournal of Probability and Statistics, 2015
- Transformation-based nonparametric estimation of multivariate densitiesJournal of Multivariate Analysis, 2015
- A Histogram Transform for ProbabilityDensity Function EstimationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Multivariate Density Estimation by Bayesian Sequential PartitioningJournal of the American Statistical Association, 2013
- Information divergence estimation based on data-dependent partitionsJournal of Statistical Planning and Inference, 2010
- Tailored randomized block MCMC methods with application to DSGE modelsJournal of Econometrics, 2010
- Encyclopedia of mathematicsChoice Reviews Online, 2006
- Nonparametric discriminant analysis and nearest neighbor classificationPattern Recognition Letters, 2003
- Consistency of data-driven histogram methods for density estimation and classificationThe Annals of Statistics, 1996
- Sequential Imputations and Bayesian Missing Data ProblemsJournal of the American Statistical Association, 1994