A Communication-Efficient Distributed Algorithm for Kernel Principal Component Analysis
Preprint
Abstract
Principal Component Analysis (PCA) is a fundamental technology in machine learning. Nowadays many high-dimension large datasets are acquired in a distributed manner, which precludes the use of centralized PCA due to the high communication cost and privacy risk. Thus, many distributed PCA algorithms are proposed, most of which, however, focus on linear cases. To efficiently extract non-linear features, this paper proposes a communication-efficient distributed kernel PCA algorithm, DKPCA, where linear and RBF kernels are applied. The key is to estimate the global empirical kernel matrix from the eigenvectors of local kernel matrices. The approximation error of the estimators is theoretically analyzed for both linear and RBF kernels. The result suggests that when eigenvalues decay fast, which is common for RBF kernels, the proposed algorithm gives high quality results with low communication cost. For data maldistribution cases, we propose a self-adaptive strategy to reduce the communication cost without compromising the accuracy. Results of simulation experiments verify our theory analysis and the classification experiments on GSE2187 dataset show the effectiveness of the proposed algorithm in practice.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Distributed learning of deep neural network over multiple agentsJournal of Network and Computer Applications, 2018
- Distributed adaptive estimation of covariance matrix eigenvectors in wireless sensor networks with application to distributed PCASignal Processing, 2014
- A useful variant of the Davis–Kahan theorem for statisticiansBiometrika, 2014
- Two-stage image denoising by principal component analysis with local pixel groupingPattern Recognition, 2010
- Moving window kernel PCA for adaptive monitoring of nonlinear processesChemometrics and Intelligent Laboratory Systems, 2009
- An Improved Algorithm for Kernel Principal Component AnalysisNeural Processing Letters, 2005
- Nonlinear dynamic process monitoring based on dynamic kernel PCAChemical Engineering Science, 2004
- Kernel PCA for Feature Extraction and De-Noising in Nonlinear RegressionNeural Computing and Applications, 2001
- Distributed Clustering Using Collective Principal Component AnalysisKnowledge and Information Systems, 2001
- Topics in Matrix Analysis1991