Variational Autoencoders Pursue PCA Directions (by Accident)
Authors:
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12398-12407
Abstract
The Variational Autoencoder (VAE) is a powerful architecture capable of representation learning and generative modeling. When it comes to learning interpretable (disentangled) representations, VAE and its variants show unparalleled performance. However, the reasons for this are unclear, since a very particular alignment of the latent embedding is needed but the design of the VAE does not encourage it in any explicit way. We address this matter and offer the following explanation: the diagonal approximation in the encoder together with the inherent stochasticity force local orthogonality of the decoder. The local behavior of promoting both reconstruction and orthogonality matches closely how the PCA embedding is chosen. Alongside providing an intuitive understanding, we justify the statement with full theoretical analysis as well as with experiments.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Deep predictive policy training using reinforcement learning2017
- Deep Feature Consistent Variational Autoencoder2017
- Stable reinforcement learning with autoencoders for tactile and visual data2016
- Modeling and Transforming Speech Using Variational Autoencoders2016
- Deep Learning Face Attributes in the Wild2015
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Learning Factorial Codes by Predictability MinimizationNeural Computation, 1992
- Calculating the Singular Values and Pseudo-Inverse of a MatrixJournal of the Society for Industrial and Applied Mathematics Series B Numerical Analysis, 1965