Learning deep representations by mutual information estimation and maximization

Authors:
  • AFAlex Fedorov
    Alex Fedorov
  • SLSamuel Lavoie-Marchildon
    Samuel Lavoie-Marchildon
  • KGKaran Grewal
    Karan Grewal
  • PBPhil Bachman
    Phil Bachman
  • ATAdam Trischler
    Adam Trischler
  • YBYoshua Bengio
    Yoshua Bengio

Abstract
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.

This publication has 9 references indexed in Scilit: