A Comparison of Variational Bounds for the Information Bottleneck Functional
Open Access
Abstract
In this short note, we relate the variational bounds proposed in Alemi et al. (2017) and Fischer (2020) for the information bottleneck (IB) and the conditional entropy bottleneck (CEB) functional, respectively. Although the two functionals were shown to be equivalent, it was empirically observed that optimizing bounds on the CEB functional achieves better generalization performance and adversarial robustness than optimizing those on the IB functional. This work tries to shed light on this issue by showing that, in the most general setting, no ordering can be established between these variational bounds, while such an ordering can be enforced by restricting the feasible sets over which the optimizations take place. The absence of such an ordering in the general setup suggests that the variational bound on the CEB functional is either more amenable to optimization or a relevant cost function for optimization in its own regard, i.e., without justification from the IB or CEB functionals.Keywords
Funding Information
- Electronic Components and Systems for European Leadership (783163)
This publication has 7 references indexed in Scilit:
- CEB Improves Model RobustnessEntropy, 2020
- The Conditional Entropy BottleneckEntropy, 2020
- On the Difference between the Information Bottleneck and the Deep Information BottleneckEntropy, 2020
- Nonlinear Information BottleneckEntropy, 2019
- Learning Representations for Neural Network-Based Classification Using the Information Bottleneck PrincipleIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Information Dropout: Learning Optimal Representations Through Noisy ComputationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- The common information of two dependent random variablesIEEE Transactions on Information Theory, 1975