Information Bottleneck and its Applications in Deep Learning
Preprint
Abstract
Information Theory (IT) has been used in Machine Learning (ML) from early days of this field. In the last decade, advances in Deep Neural Networks (DNNs) have led to surprising improvements in many applications of ML. The result has been a paradigm shift in the community toward revisiting previous ideas and applications in this new framework. Ideas from IT are no exception. One of the ideas which is being revisited by many researchers in this new era, is Information Bottleneck (IB); a formulation of information extraction based on IT. The IB is promising in both analyzing and improving DNNs. The goal of this survey is to review the IB concept and demonstrate its applications in deep learning. The information theoretic nature of IB, makes it also a good candidate in showing the more general concept of how IT can be used in ML. Two important concepts are highlighted in this narrative on the subject, i) the concise and universal view that IT provides on seemingly unrelated methods of ML, demonstrated by explaining how IB relates to minimal sufficient statistics, stochastic gradient descent, and variational auto-encoders, and ii) the common technical mistakes and problems caused by applying ideas from IT, which is discussed by a careful study of some recent methods suffering from them.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Entropy-SGD: biasing gradient descent into wide valleys*Journal of Statistical Mechanics: Theory and Experiment, 2019
- Flow of Information in Feed-Forward Denoising Neural Networks2018
- Emergence of Invariance and Disentanglement in Deep Representations2018
- Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks2018
- Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A reviewInternational Journal of Automation and Computing, 2017
- Deep learning and the information bottleneck principle2015
- Learning and generalization with the information bottleneckTheoretical Computer Science, 2010
- An algorithm for computing the capacity of arbitrary discrete memoryless channelsIEEE Transactions on Information Theory, 1972
- The sizes of compact subsets of Hilbert space and continuity of Gaussian processesJournal of Functional Analysis, 1967
- On the Shannon theory of information transmission in the case of continuous signalsIRE Transactions on Information Theory, 1956