Data Efficiency, Dimensionality Reduction, and the Generalized Symmetric Information Bottleneck
- 7 June 2024
- journal article
- research article
- Published by MIT Press in Neural Computation
- Vol. 36 (7) , 1353-1379
Abstract
The symmetric information bottleneck (SIB), an extension of the more familiar information bottleneck, is a dimensionality-reduction technique that simultaneously compresses two random variables to preserve information between their compressed versions. We introduce the generalized symmetric information bottleneck (GSIB), which explores different functional forms of the cost of such simultaneous reduction. We then explore the data set size requirements of such simultaneous compression. We do this by deriving bounds and root-mean-squared estimates of statistical fluctuations of the involved loss functions. We show that in typical situations, the simultaneous GSIB compression requires qualitatively less data to achieve the same errors compared to compressing variables one at a time. We suggest that this is an example of a more general principle that simultaneous compression is more data efficient than independent compression of each of the input variables.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Using the right tool for the job: the difference between unsupervised and supervised analyses of multivariate ecological dataOecologia, 2021
- A technical review of canonical correlation analysis for neuroscience applicationsHuman Brain Mapping, 2020
- Neuroscience Needs Behavior: Correcting a Reductionist BiasNeuron, 2017
- A Bayesian supervised dual‐dimensionality reduction model for simultaneous decoding of LFP and spike train signalsStat, 2017
- The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information RepositoryJournal of Digital Imaging, 2013
- Variable selection in regression—a tutorialJournal of Chemometrics, 2010
- Learning and generalization with the information bottleneckTheoretical Computer Science, 2010
- Convergence properties of functional estimates for discrete distributionsRandom Structures & Algorithms, 2001
- Multidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric HypothesisPsychometrika, 1964
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948