Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal
Open Access
Abstract
The problem of how to properly quantify redundant information is an open question that has been the subject of much recent research. Redundant information refers to information about a target variable S that is common to two or more predictor variables . It can be thought of as quantifying overlapping information content or similarities in the representation of S between the . We present a new measure of redundancy which measures the common change in surprisal shared between variables at the local or pointwise level. We provide a game-theoretic operational definition of unique information, and use this to derive constraints which are used to obtain a maximum entropy distribution. Redundancy is then calculated from this maximum entropy distribution by counting only those local co-information terms which admit an unambiguous interpretation as redundant information. We show how this redundancy measure can be used within the framework of the Partial Information Decomposition (PID) to give an intuitive decomposition of the multivariate mutual information into redundant, unique and synergistic contributions. We compare our new measure to existing approaches over a range of example systems, including continuous Gaussian variables. Matlab code for the measure is provided, including all considered examples.
Keywords
All Related Versions
Funding Information
- Wellcome Trust (107801/Z/15/Z)
- MURI/EPSRC (EP/N019261/1)
This publication has 44 references indexed in Scilit:
- Partial information decomposition as a unified approach to the specification of neural goal functionsBrain and Cognition, 2017
- A statistical framework for neuroimaging data analysis based on mutual information estimated via a gaussian copulaHuman Brain Mapping, 2016
- Information Decomposition and SynergyEntropy, 2015
- Quantifying Unique InformationEntropy, 2014
- Intersection Information Based on Common RandomnessEntropy, 2014
- A Framework for the Local Information Dynamics of Distributed Computation in Complex Systems2014
- Synergy, redundancy, and multivariate information measures: an experimentalist’s perspectiveJournal of Computational Neuroscience, 2013
- Bivariate measure of redundant informationPhysical Review E, 2013
- Shared Information—New Insights and Problems in Decomposing Information in Complex Systems2013
- A Mathematical Theory of CommunicationBell System Technical Journal, 1948