Marc Mézard
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Marc Mézard has a total of 42 co-authors. Here are the top co-authors by number of shared publications:
- Florent Krzakala
École Polytechnique Fédérale de Lausanne (EPFL) IdePHICS Lab CH-1015 Lausanne Switzerland
4 shared publications - Francesco Camilli
The Abdus Salam International Centre for Theoretical Physics, Trieste, Italy
3 shared publications - Carlo Lucibello
Bocconi University
2 shared publications - Lenka Zdeborová
Institut de Physique Théorique, CNRS, CEA, Université Paris-Saclay, 91191 Gif-sur-Yvette, France
2 shared publications - Riccardo Zecchina
Bocconi University
2 shared publications - Tony Bonnaire
Laboratoire de Physique de l'École Normale Supérieure ENS Université PSL CNRS Sorbonne Université Université Paris Cité F-75005 Paris France
2 shared publications
Latest Publications
- Open Access
- 2 September 2026
- Published by Springer Nature
- p. 21-29
Controlling AI? A Technological Revolution Waiting for a Theory
- 2 September 2026
- Published by Springer Nature
- p. 21-29
Open AccessControlling AI? A Technological Revolution Waiting for a Theory
- Open Access
- 1 August 2026
- Published by IOP Publishing in Journal of Statistical Mechanics: Theory and Experiment
- Vol. 2026 (8) , 084007
Why diffusion models do not memorize: the role of implicit dynamical regularization in training
Diffusion models have attained remarkable success across a wide range of generative tasks. A key challenge lies in understanding the mechanisms that prevent their memorization of training data and allow generalization. In this work, we investigate the role of the training dynamics in the transition from generalization to memorization. Through extensive experiments and theoretical analysis, we identify two distinct timescales: an early time τ gen at which models begin to generate high-quality samples and a later time τ mem beyond which memorization emerges. Crucially, we found that τ mem increases linearly with the training set size n , whereas τ gen remains constant. This creates a growing window of training times with n where models generalize effectively, despite showing strong memorization if training continues beyond it. It is only when n becomes larger than a model-dependent threshold that overfitting disappears at infinite training times. These findings reveal a form of implicit dynamical regularization in the training dynamics, which allows to avoid memorization even in highly overparameterized settings. Our findings are supported by numerical experiments with standard U-Net architectures on realistic and synthetic datasets, alongside a theoretical analysis using a tractable random features model studied in the high-dimensional limit. 4 4 https://github.com/tbonnair/Why-Diffusion-Models-Don-t-Memorize .
- 1 August 2026
- Published by IOP Publishing in Journal of Statistical Mechanics: Theory and Experiment
- Vol. 2026 (8) , 084007
Open AccessWhy diffusion models do not memorize: the role of implicit dynamical regularization in training
Diffusion models have attained remarkable success across a wide range of generative tasks. A key challenge lies in understanding the mechanisms that prevent their memorization of training data and allow generalization. In this work, we investigate the role of the training dynamics in the transition from generalization to memorization. Through extensive experiments and theoretical analysis, we identify two distinct timescales: an early time τ gen at which models begin to generate high-quality samples and a later time τ mem beyond which memorization emerges. Crucially, we found that τ mem increases linearly with the training set size n , whereas τ gen remains constant. This creates a growing window of training times with n where models generalize effectively, despite showing strong memorization if training continues beyond it. It is only when n becomes larger than a model-dependent threshold that overfitting disappears at infinite training times. These findings reveal a form of implicit dynamical regularization in the training dynamics, which allows to avoid memorization even in highly overparameterized settings. Our findings are supported by numerical experiments with standard U-Net architectures on realistic and synthetic datasets, alongside a theoretical analysis using a tractable random features model studied in the high-dimensional limit. 4 4 https://github.com/tbonnair/Why-Diffusion-Models-Don-t-Memorize .
- Open Access
- 11 June 2026
- Published by American Physical Society (APS) in Physical Review Research
- Vol. 8 (2)
Dynamical learning in deep asymmetric recurrent neural networks
We investigate recurrent neural networks with asymmetric interactions and demonstrate that the inclusion of self-couplings or sparse excitatory intermodule connections leads to the emergence of a densely connected manifold of dynamically accessible stable configurations. This representation manifold is exponentially large in system size and is reachable through simple local dynamics, despite constituting a subdominant subset of the global configuration space. We further show that learning can be implemented directly on this structure via a fully local, gradient-free mechanism that selectively stabilizes a single task-relevant network configuration. Unlike error-driven or contrastive learning schemes, this approach does not require explicit comparisons between network states obtained with and without output supervision. Instead, transient supervisory signals bias the dynamics toward the representation manifold, after which local plasticity consolidates the attained configuration, effectively shaping the latent representation space. Numerical evaluations on standard image classification benchmarks indicate performance comparable to that of multilayer perceptrons trained using backpropagation. More generally, these results suggest that the dynamical accessibility of fixed points and the stabilization of internal network dynamics constitute viable alternative principles for learning in recurrent systems, with conceptual links to statistical physics and potential implications for biologically motivated and neuromorphic computing architectures.
- 11 June 2026
- Published by American Physical Society (APS) in Physical Review Research
- Vol. 8 (2)
Open AccessDynamical learning in deep asymmetric recurrent neural networks
We investigate recurrent neural networks with asymmetric interactions and demonstrate that the inclusion of self-couplings or sparse excitatory intermodule connections leads to the emergence of a densely connected manifold of dynamically accessible stable configurations. This representation manifold is exponentially large in system size and is reachable through simple local dynamics, despite constituting a subdominant subset of the global configuration space. We further show that learning can be implemented directly on this structure via a fully local, gradient-free mechanism that selectively stabilizes a single task-relevant network configuration. Unlike error-driven or contrastive learning schemes, this approach does not require explicit comparisons between network states obtained with and without output supervision. Instead, transient supervisory signals bias the dynamics toward the representation manifold, after which local plasticity consolidates the attained configuration, effectively shaping the latent representation space. Numerical evaluations on standard image classification benchmarks indicate performance comparable to that of multilayer perceptrons trained using backpropagation. More generally, these results suggest that the dynamical accessibility of fixed points and the stabilization of internal network dynamics constitute viable alternative principles for learning in recurrent systems, with conceptual links to statistical physics and potential implications for biologically motivated and neuromorphic computing architectures.
- Open Access
- 5 June 2026
- Published by American Physical Society (APS) in Physical Review X
- Vol. 16 (2)
Are Neural Networks Collision Resistant?
Collision-resistant hash functions are a fundamental cryptographic primitive that rely on the computational hardness of finding two inputs that produce the same output. Motivated by this problem, we study the complexity of finding collisions in a family of neural networks with oscillating activation functions. A neural network trained on a classification task is specified by a set of weights assigning a label to each data point, and a collision is defined as two distinct weight configurations that produce the same labeling. We show that, within this class of neural networks, the space of collisions exhibits an overlap gap property, whereby certain overlap values between distinct solutions are forbidden. This property is a geometric feature of the solution landscape in high-dimensional random constraint satisfaction problems that has recently emerged as a powerful indicator of algorithmic barriers. Our analysis predicts a regime in which efficient algorithms fail to find collisions. This prediction is supported by numerical experiments using approximate message passing algorithms, which cease to return collisions well below the threshold predicted by theory. Neural networks, therefore, provide a class of candidate collision-resistant functions that, for suitable parameter choices, depart from existing constructions based on lattices. Beyond their cryptographic relevance, our results reveal forms of computational hardness in large neural networks that may be of independent interest.
- 5 June 2026
- Published by American Physical Society (APS) in Physical Review X
- Vol. 16 (2)
Open AccessAre Neural Networks Collision Resistant?
Collision-resistant hash functions are a fundamental cryptographic primitive that rely on the computational hardness of finding two inputs that produce the same output. Motivated by this problem, we study the complexity of finding collisions in a family of neural networks with oscillating activation functions. A neural network trained on a classification task is specified by a set of weights assigning a label to each data point, and a collision is defined as two distinct weight configurations that produce the same labeling. We show that, within this class of neural networks, the space of collisions exhibits an overlap gap property, whereby certain overlap values between distinct solutions are forbidden. This property is a geometric feature of the solution landscape in high-dimensional random constraint satisfaction problems that has recently emerged as a powerful indicator of algorithmic barriers. Our analysis predicts a regime in which efficient algorithms fail to find collisions. This prediction is supported by numerical experiments using approximate message passing algorithms, which cease to return collisions well below the threshold predicted by theory. Neural networks, therefore, provide a class of candidate collision-resistant functions that, for suitable parameter choices, depart from existing constructions based on lattices. Beyond their cryptographic relevance, our results reveal forms of computational hardness in large neural networks that may be of independent interest.
- Open Access
- 4 March 2025
- Published by MathDoc/Centre Mersenne in Comptes Rendus. Physique
- Vol. 26 (G1) , 253-257
La vie et l’œuvre de Gérard Toulouse
- 4 March 2025
- Published by MathDoc/Centre Mersenne in Comptes Rendus. Physique
- Vol. 26 (G1) , 253-257
Open AccessLa vie et l’œuvre de Gérard Toulouse
- Open Access
Dynamical Regimes of Diffusion Models
Using statistical physics methods, we study generative diffusion models in the regime where the dimension of space and the number of data are large, and the score function has been trained optimally. Our analysis reveals three distinct dynamical regimes during the backward generative diffusion process. The generative dynamics, starting from pure noise, encounters first a 'speciation' transition where the gross structure of data is unraveled, through a mechanism similar to symmetry breaking in phase transitions. It is followed at later time by a 'collapse' transition where the trajectories of the dynamics become attracted to one of the memorized data points, through a mechanism which is similar to the condensation in a glass phase. For any dataset, the speciation time can be found from a spectral analysis of the correlation matrix, and the collapse time can be found from the estimation of an 'excess entropy' in the data. The dependence of the collapse time on the dimension and number of data provides a thorough characterization of the curse of dimensionality for diffusion models. Analytical solutions for simple models like high-dimensional Gaussian mixtures substantiate these findings and provide a theoretical framework, while extensions to more complex scenarios and numerical validations with real datasets confirm the theoretical predictions.
Open AccessDynamical Regimes of Diffusion Models
Using statistical physics methods, we study generative diffusion models in the regime where the dimension of space and the number of data are large, and the score function has been trained optimally. Our analysis reveals three distinct dynamical regimes during the backward generative diffusion process. The generative dynamics, starting from pure noise, encounters first a 'speciation' transition where the gross structure of data is unraveled, through a mechanism similar to symmetry breaking in phase transitions. It is followed at later time by a 'collapse' transition where the trajectories of the dynamics become attracted to one of the memorized data points, through a mechanism which is similar to the condensation in a glass phase. For any dataset, the speciation time can be found from a spectral analysis of the correlation matrix, and the collapse time can be found from the estimation of an 'excess entropy' in the data. The dependence of the collapse time on the dimension and number of data provides a thorough characterization of the curse of dimensionality for diffusion models. Analytical solutions for simple models like high-dimensional Gaussian mixtures substantiate these findings and provide a theoretical framework, while extensions to more complex scenarios and numerical validations with real datasets confirm the theoretical predictions.
- Top Cited Papers
- 13 February 2024
- Published by American Physical Society (APS) in Physical Review Letters
- Vol. 132 (7) , 077301
Exponential Capacity of Dense Associative Memories
Recent generalizations of the Hopfield model of associative memories are able to store a number of random patterns that grows exponentially with the number of neurons, . Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architecture widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, , and lower bounds for the maximum of the load for which all patterns can be retrieved, , as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams.- 13 February 2024
- Published by American Physical Society (APS) in Physical Review Letters
- Vol. 132 (7) , 077301
Top Cited PapersExponential Capacity of Dense Associative Memories
Recent generalizations of the Hopfield model of associative memories are able to store a number of random patterns that grows exponentially with the number of neurons, . Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architecture widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, , and lower bounds for the maximum of the load for which all patterns can be retrieved, , as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams. - Open Access
- 12 February 2024
- Published by IOP Publishing in Journal of Physics A: Mathematical and Theoretical
- Vol. 57 (8) , 085002
The decimation scheme for symmetric matrix factorization
Matrix factorization is an inference problem that has acquired importance due to its vast range of applications that go from dictionary learning to recommendation systems and machine learning with deep networks. The study of its fundamental statistical limits represents a true challenge, and despite a decade-long history of efforts in the community, there is still no closed formula able to describe its optimal performances in the case where the rank of the matrix scales linearly with its size. In the present paper, we study this extensive rank problem, extending the alternative ‘decimation’ procedure that was recently introduced by Camilli and Mézard, and carry out a thorough study of its performance. Decimation aims at recovering one column/line of the factors at a time, by mapping the problem into a sequence of neural network models of associative memory at a tunable temperature. The main advantage of decimation is that its theoretical performance can be studied using statistical physics techniques. In this paper we provide the general analysis applying to a large class of compactly supported priors and we show that the replica symmetric free entropy of the neural network models takes a universal form in the low temperature limit. We then study the decimation phase diagrams in two concrete cases, the sparse Ising prior and the uniform prior, for which we show that matrix factorization is theoretically possible when the ration P / N of rank to variable is below a certain threshold. This suggest that a possible route to solving algorithmically the matrix factorization problem could be to find efficient decimation algorithms; in this respect, we show that simple simulated annealing, though being effective for limited signal sizes, is not scalable.
- 12 February 2024
- Published by IOP Publishing in Journal of Physics A: Mathematical and Theoretical
- Vol. 57 (8) , 085002
Open AccessThe decimation scheme for symmetric matrix factorization
Matrix factorization is an inference problem that has acquired importance due to its vast range of applications that go from dictionary learning to recommendation systems and machine learning with deep networks. The study of its fundamental statistical limits represents a true challenge, and despite a decade-long history of efforts in the community, there is still no closed formula able to describe its optimal performances in the case where the rank of the matrix scales linearly with its size. In the present paper, we study this extensive rank problem, extending the alternative ‘decimation’ procedure that was recently introduced by Camilli and Mézard, and carry out a thorough study of its performance. Decimation aims at recovering one column/line of the factors at a time, by mapping the problem into a sequence of neural network models of associative memory at a tunable temperature. The main advantage of decimation is that its theoretical performance can be studied using statistical physics techniques. In this paper we provide the general analysis applying to a large class of compactly supported priors and we show that the replica symmetric free entropy of the neural network models takes a universal form in the low temperature limit. We then study the decimation phase diagrams in two concrete cases, the sparse Ising prior and the uniform prior, for which we show that matrix factorization is theoretically possible when the ration P / N of rank to variable is below a certain threshold. This suggest that a possible route to solving algorithmically the matrix factorization problem could be to find efficient decimation algorithms; in this respect, we show that simple simulated annealing, though being effective for limited signal sizes, is not scalable.
- Open Access
The Exponential Capacity of Dense Associative Memories
Recent generalizations of the Hopfield model of associative memories are able to store a number $P$ of random patterns that grows exponentially with the number $N$ of neurons, $P=\exp(αN)$. Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architectures widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, $α_1$, and lower bounds for the maximum of the load $α$ for which all patterns can be retrieved, $α_c$, as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams.Open AccessThe Exponential Capacity of Dense Associative Memories
Recent generalizations of the Hopfield model of associative memories are able to store a number $P$ of random patterns that grows exponentially with the number $N$ of neurons, $P=\exp(αN)$. Besides the huge storage capacity, another interesting feature of these networks is their connection to the attention mechanism which is part of the Transformer architectures widely applied in deep learning. In this work, we study a generic family of pattern ensembles using a statistical mechanics analysis which gives exact asymptotic thresholds for the retrieval of a typical pattern, $α_1$, and lower bounds for the maximum of the load $α$ for which all patterns can be retrieved, $α_c$, as well as sizes of attraction basins. We discuss in detail the cases of Gaussian and spherical patterns, and show that they display rich and qualitatively different phase diagrams. - Open Access
- 10 December 2023
- Published by Springer Nature in Indian Journal of Physics
- Vol. 98 (11) , 3757-3768
Spin glass theory and its new challenge: structured disorder
This paper first describes, from a high-level viewpoint, the main challenges that had to be solved in order to develop a theory of spin glasses in the last fifty years. It then explains how important inference problems, notably those occurring in machine learning, can be formulated as problems in statistical physics of disordered systems. However, the main questions that we face in the analysis of deep networks require to develop a new chapter of spin glass theory, which will address the challenge of structured data.
- 10 December 2023
- Published by Springer Nature in Indian Journal of Physics
- Vol. 98 (11) , 3757-3768
Open AccessSpin glass theory and its new challenge: structured disorder
This paper first describes, from a high-level viewpoint, the main challenges that had to be solved in order to develop a theory of spin glasses in the last fifty years. It then explains how important inference problems, notably those occurring in machine learning, can be formulated as problems in statistical physics of disordered systems. However, the main questions that we face in the analysis of deep networks require to develop a new chapter of spin glass theory, which will address the challenge of structured data.