Learning deep representations by mutual information estimation and maximization
Authors:
- R Devon HjelmRHR Devon Hjelm
- Alex FedorovAFAlex Fedorov
- Samuel Lavoie-MarchildonSLSamuel Lavoie-Marchildon
- Karan GrewalKGKaran Grewal
- Phil BachmanPBPhil Bachman
- Adam TrischlerATAdam Trischler
- Yoshua BengioYBYoshua Bengio
Preprint
Abstract
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency2018
- Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Training Products of Experts by Minimizing Contrastive DivergenceNeural Computation, 2002
- Slow Feature Analysis: Unsupervised Learning of InvariancesNeural Computation, 2002
- Independent component analysis: algorithms and applicationsNeural Networks, 2000
- Nonlinear independent component analysis: Existence and uniqueness resultsNeural Networks, 1999
- Learning Factorial Codes by Predictability MinimizationNeural Computation, 1992
- Self-organization in a perceptual networkComputer, 1988
- Asymptotic evaluation of certain markov process expectations for large time. IVCommunications on Pure and Applied Mathematics, 1983