Measuring multivariate redundant information with pointwise common change in surprisal
Preprint
Abstract
The problem of how to properly quantify redundant information is an open question that has been the subject of much recent research. Redundant information refers to information about a target variable S that is common to two or more predictor variables Xi. It can be thought of as quantifying overlapping information content or similarities in the representation of S between the Xi. We present a new measure of redundancy which measures the common change in surprisal shared between variables at the local or pointwise level. We demonstrate how this redundancy measure can be used within the framework of the Partial Information Decomposition (PID) to give an intuitive decomposition of the multivariate mutual information for a range of example systems, including continuous Gaussian variables. We also propose a modification of the PID in which we normalise partial information terms from non-disjoint sets of sources within the same level of the redundancy lattice, to prevent negative terms resulting from over-counting dependent partial information values. Our redundancy measure is easy to compute, and Matlab code implementing the measure, together with all considered examples, is provided.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Partial information decomposition as a unified approach to the specification of neural goal functionsBrain and Cognition, 2017
- Characterizing the dynamics of mental representations: the temporal generalization methodTrends in Cognitive Sciences, 2014
- Quantifying Synergistic Mutual Information2014
- A Framework for the Local Information Dynamics of Distributed Computation in Complex Systems2014
- Synergy, redundancy, and multivariate information measures: an experimentalist’s perspectiveJournal of Computational Neuroscience, 2013
- Shared Information—New Insights and Problems in Decomposing Information in Complex Systems2013
- A novel test to determine the significance of neural selectivity to single and multiple potentially correlated stimulus featuresJournal of Neuroscience Methods, 2012
- Extracting information from neuronal populations: information theory and decoding approachesNature Reviews Neuroscience, 2009
- Multiple mutual informations and multiple interactions in frequency dataInformation and Control, 1980
- Multivariate Information TransmissionPsychometrika, 1954