Scalable and Flexible Multiview MAX-VAR Canonical Correlation Analysis
Open Access
- 26 April 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Signal Processing
- Vol. 65 (16) , 4150-4165
Abstract
Generalized canonical correlation analysis (GCCA) aims at finding latent low-dimensional common structure from multiple views (feature vectors in different domains) of the same entities. Unlike principal component analysis that handles a single view, (G)CCA is able to integrate information from different feature spaces. Here we focus on MAX-VAR GCCA, a popular formulation that has recently gained renewed interest in multilingual processing and speech modeling. The classic MAX-VAR GCCA problem can be solved optimally via eigen-decomposition of a matrix that compounds the (whitened) correlation matrices of the views; but this solution has serious scalability issues, and is not directly amenable to incorporating pertinent structural constraints such as nonnegativity and sparsity on the canonical components. We posit regularized MAX-VAR GCCA as a nonconvex optimization problem and propose an alternating optimization-based algorithm to handle it. Our algorithm alternates between inexact solutions of a regularized least squares subproblem and a manifold-constrained nonconvex subproblem, thereby achieving substantial memory and computational savings. An important benefit of our design is that it can easily handle structure-promoting regularization. We show that the algorithm globally converges to a critical point at a sublinear rate, and approaches a global optimal solution at a linear rate when no regularization is considered. Judiciously designed simulations and large-scale word embedding tasks are employed to showcase the effectiveness of the proposed algorithm.Keywords
All Related Versions
Funding Information
- National Science Foundation (NSF ECCS-1608961, NSF IIS-1447788)
- National Science Foundation (CCF-1526078)
- Air Force Office of Scientific Research (15RT0767)
This publication has 30 references indexed in Scilit:
- A Globally Convergent Algorithm for Nonconvex Optimization Based on Block Coordinate UpdateJournal of Scientific Computing, 2017
- Multiview LSA: Representation Learning via Generalized CCA2015
- Community Evaluation and Exchange of Word Vectors at wordvectors.org2014
- A Block Coordinate Descent Method for Regularized Multiconvex Optimization with Applications to Nonnegative Tensor Factorization and CompletionSIAM Journal on Imaging Sciences, 2013
- A Unified Convergence Analysis of Block Successive Minimization Methods for Nonsmooth OptimizationSIAM Journal on Optimization, 2013
- Distributed Optimization and Statistical Learning via the Alternating Direction Method of MultipliersFoundations and Trends in Machine Learning, 2011
- Predicting Human Brain Activity Associated with the Meanings of NounsScience, 2008
- Canonical Analysis of Several Sets of VariablesBiometrika, 1971
- A Generalized Solution of the Orthogonal Procrustes ProblemPsychometrika, 1966
- Generalized canonical correlations and their applications to experimental dataJournal of Clinical Psychology, 1961