The Stochastic complexity of spin models: how simple are simple spin models?
Authors:
- Alberto BerettaABAlberto Beretta
- Claudia BattistinCBClaudia Battistin
- Clélia de MulatierCMClélia de Mulatier
- Iacopo MastromatteoIMIacopo Mastromatteo
- Matteo MarsiliMMMatteo Marsili
Preprint
Abstract
Models can be simple for different reasons. In science, models are simple when they depend on few variables, they are meant to yield predictions that can be falsified. Parameters are often mutually constrained within scales. They spot symmetries and conservation laws specific of the phenomenon under study. Models in statistical learning are simple because they are easy to evaluate, train and/or to infer. They are simple to interpret in terms of low order dependencies (e.g. graphical models). Their aim is not to uncover fundamental laws but to "generalise well", i.e. to describe well yet unseen data. We show that, for spin models with interactions of arbitrary order, the information theoretic notion of complexity (or simplicity) provided by Minimum Description Length (MDL) conforms with the scientific notion of simplicity rather than with that in machine learning. Simple models in MDL have statistical dependencies concentrated in groups of few variables, they afford predictions on independencies that are easy to falsify. On the contrary, pairwise interacting models, which are often used in statistical learning, appear to be rather complex.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Emergence of Compositional Representations in Restricted Boltzmann MachinesPhysical Review Letters, 2017
- Sparse model selection in the highly under-sampled regimeJournal of Statistical Mechanics: Theory and Experiment, 2016
- On the Sufficiency of Pairwise Interactions in Maximum Entropy Models of NetworksJournal of Statistical Physics, 2016
- Inferring biological tasks using Pareto analysis of high-dimensional dataNature Methods, 2015
- Cluster variation method in statistical physics and probabilistic graphical modelsJournal of Physics A: General Physics, 2005
- Simplicity: a unifying principle in cognitive science?Trends in Cognitive Sciences, 2003
- Counting probability distributions: Differential geometry and model selectionProceedings of the National Academy of Sciences of the United States of America, 2000
- Modeling by shortest data descriptionAutomatica, 1978
- The unreasonable effectiveness of mathematics in the natural sciences. Richard courant lecture in mathematical sciences delivered at New York University, May 11, 1959Communications on Pure and Applied Mathematics, 1960
- An invariant form for the prior probability in estimation problemsProceedings of the Royal Society of London. A. Mathematical and Physical Sciences, 1946