Data efficiency, dimensionality reduction, and the generalized symmetric information bottleneck
Authors:
Preprint
Abstract
The Symmetric Information Bottleneck (SIB), an extension of the more familiar Information Bottleneck, is a dimensionality reduction technique that simultaneously compresses two random variables to preserve information between their compressed versions. We introduce the Generalized Symmetric Information Bottleneck (GSIB), which explores different functional forms of the cost of such simultaneous reduction. We then explore the dataset size requirements of such simultaneous compression. We do this by deriving bounds and root-mean-squared estimates of statistical fluctuations of the involved loss functions. We show that, in typical situations, the simultaneous GSIB compression requires qualitatively less data to achieve the same errors compared to compressing variables one at a time. We suggest that this is an example of a more general principle that simultaneous compression is more data efficient than independent compression of each of the input variables.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Using the right tool for the job: the difference between unsupervised and supervised analyses of multivariate ecological dataOecologia, 2021
- A technical review of canonical correlation analysis for neuroscience applicationsHuman Brain Mapping, 2020
- A Bayesian supervised dual‐dimensionality reduction model for simultaneous decoding of LFP and spike train signalsStat, 2017
- Variable selection in regression—a tutorialJournal of Chemometrics, 2010
- Learning and generalization with the information bottleneckTheoretical Computer Science, 2010
- PLS-regression: a basic tool of chemometricsChemometrics and Intelligent Laboratory Systems, 2001
- Convergence properties of functional estimates for discrete distributionsRandom Structures & Algorithms, 2001
- Estimating the errors on measured entropy and mutual informationPhysica D: Nonlinear Phenomena, 1999
- Multidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric HypothesisPsychometrika, 1964
- Analysis of a complex of statistical variables into principal components.Journal of Educational Psychology, 1933