ASAntonio Sclocchi

Antonio Sclocchi

*This is an algorithmically generated author record based on publicly available information.

Antonio Sclocchi has a total of 11 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • A new critical phase in jammed models : jamming is even cooler than before

    by Antonio Sclocchi

    Une nouvelle phase critique pour les modèles brouillés : le brouillage ("jamming") est encore plus cool qu’avant: Au cours des vingt dernières années, un courant de recherche important a caractérisé la transition de brouillage, un point critique à température zéro des systèmes ayant des interactions répulsives à courte portée. Plusieurs de ses propriétés sont indépendantes de la dimension spatiale, avec des prédictions de champ moyen étant valables même pour les systèmes bidimensionnels. Dans cette thèse, nous montrons l’existence de ce comportement critique du point de brouillage ("jamming") dans toute une phase brouillée ("jammed"). Ce comportement est observé en étudiant le potentiel de répulsion linéaire dans les sphères molles et le modèle de champ moyen correspondant, c’est-à-dire le perceptron. Nous montrons que le point non différentiable du potentiel d’interaction produit un réseau de contact de sphères tangentes à chaque densité surcomprimée, même loin de la transition de jamming.Les réseaux de contact caractérisant les minima du système sont isostatiques, critiquement auto-organisés et marginalement stables. Dans la première partie, nous utilisons la théorie du champ moyen pour le cas du perceptron et nous la validons par des simulations numériques. Ensuite, nous utilisons des simulations numériques pour étudier le cas des sphères molles en deux et trois dimensions. Dans les deux cas, nous caractérisons la phase marginalement stable et nous montrons que les exposants critiques correspondent à ceux connus pour la transition de jamming. De plus, nous définissons un protocole de compression et nous étudions numériquement les propriétés statistiques des avalanches dans la phase critique du perceptron. Nos résultats sont en accord parfait avec la théorie sous-jacente. Ce travail montre l’existence d’une phase critique dans les dimensions finies dont la classe d’universalité correspond au jamming des sphères. Cela ouvre de nouvelles perspectives pour l’étude des verres marginalement stables et des paysages énergétiques qui leur sont associés.

  • Scaling laws and representation learning in simple hierarchical languages: Transformers versus convolutional architectures

    by Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, Matthieu Wyart

    How do neural language models acquire a language's structure when trained for next-token prediction? We address this question by deriving theoretical scaling laws for neural network performance on synthetic datasets generated by the random hierarchy model (RHM)-an ensemble of probabilistic context-free grammars designed to capture the hierarchical structure of natural language while remaining analytically tractable. Previously, we developed a theory of representation learning based on data correlations that explains how deep learning models capture the hierarchical structure of the data sequentially, one layer at a time. Here, we extend our theoretical framework to account for architectural differences. In particular, we predict and empirically validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance compared to transformer models, which rely on global self-attention mechanisms. This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.


  • Open Access

    Probing the latent hierarchical structure of data via diffusion models*

    by Antonio Sclocchi, Alessandro Favero, Noam Itzhak Levi, Matthieu Wyart

    High-dimensional data must be highly structured to be learnable. Although the compositional and hierarchical nature of data is often put forward to explain learnability, quantitative measurements establishing these properties are scarce. Likewise, accessing the latent variables underlying such a data structure remains a challenge. In this work, we show that forward–backward experiments in diffusion-based models, where data is noised and then denoised to generate new samples, are a promising tool to probe the latent structure of data. We predict in simple hierarchical models that, in this process, changes in data occur by correlated chunks, with a length scale that diverges at a noise level where a phase transition is known to take place. Remarkably, we confirm this prediction in both text and image datasets using state-of-the-art diffusion models. Our results show how latent variable changes manifest in the data and establish how to measure these effects in real data using diffusion models.


  • Open Access

    Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures

    by Francesco Cagnetta, Alessandro Favero, Antonio Sclocchi, Matthieu Wyart

    How do neural language models acquire a language's structure when trained for next-token prediction? We address this question by deriving theoretical scaling laws for neural network performance on synthetic datasets generated by the Random Hierarchy Model (RHM) -- an ensemble of probabilistic context-free grammars designed to capture the hierarchical structure of natural language while remaining analytically tractable. Previously, we developed a theory of representation learning based on data correlations that explains how deep learning models capture the hierarchical structure of the data sequentially, one layer at a time. Here, we extend our theoretical framework to account for architectural differences. In particular, we predict and empirically validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance compared to transformer models, which rely on global self-attention mechanisms. This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.

  • Top Cited Papers
    Open Access

    A phase transition in diffusion models reveals the hierarchical nature of data

    by Antonio Sclocchi, Alessandro Favero, Matthieu Wyart

    Understanding the structure of real data is paramount in advancing modern deep-learning methodologies. Natural data such as images are believed to be composed of features organized in a hierarchical and combinatorial manner, which neural networks capture during learning. Recent advancements show that diffusion models can generate high-quality images, hinting at their ability to capture this underlying compositional structure. We study this phenomenon in a hierarchical generative model of data. We find that the backward diffusion process acting after a time t is governed by a phase transition at some threshold time, where the probability of reconstructing high-level features, like the class of an image, suddenly drops. Instead, the reconstruction of low-level features, such as specific details of an image, evolves smoothly across the whole diffusion process. This result implies that at times beyond the transition, the class has changed, but the generated sample may still be composed of low-level elements of the initial image. We validate these theoretical insights through numerical experiments on class-unconditional ImageNet diffusion models. Our analysis characterizes the relationship between time and scale in diffusion models and puts forward generative models as powerful tools to model combinatorial data properties.


  • Open Access

    On the different regimes of stochastic gradient descent

    by Antonio Sclocchi, Matthieu Wyart

    Modern deep networks are trained with stochastic gradient descent (SGD) whose key hyperparameters are the number of data considered at each step or batch sizeB, and the step size or learning rateη. For smallBand largeη, SGD corresponds to a stochastic evolution of the parameters, whose noise amplitude is governed by the “temperature”T≡η/B. Yet this description is observed to break down for sufficiently large batchesB≥B∗, or simplifies to gradient descent (GD) when the temperature is sufficiently small. Understanding where these cross-overs take place remains a central challenge. Here, we resolve these questions for a teacher-student perceptron classification model and show empirically that our key predictions still apply to deep networks. Specifically, we obtain a phase diagram in theB-ηplane that separates three dynamical phases: i) a noise-dominated SGD governed by temperature, ii) a large-first-step-dominated SGD and iii) GD. These different phases also correspond to different regimes of generalization error. Remarkably, our analysis reveals that the batch sizeB∗separating regimes (i) and (ii) scale with the sizePof the training set, with an exponent that characterizes the hardness of the classification problem.


  • High-dimensional optimization under nonconvex excluded volume constraints

    by Antonio Sclocchi, Pierfrancesco Urbani

    We consider high-dimensional random optimization problems where the dynamical variables are subjected to nonconvex excluded volume constraints. We focus on the case in which the cost function is a simple quadratic cost and the excluded volume constraints are modeled by a perceptron constraint satisfaction problem. We show that depending on the density of constraints, one can have different situations. If the number of constraints is small, one typically has a phase where the ground state of the cost function is unique and sits on the boundary of the island of configurations allowed by the constraints. In this case, there is a hypostatic number of marginally satisfied constraints. If the number of constraints is increased one enters a glassy phase where the cost function has many local minima sitting again on the boundary of the regions of allowed configurations. At the phase transition point, the total number of marginally satisfied constraints becomes equal to the number of degrees of freedom in the problem and therefore we say that these minima are isostatic. We conjecture that by increasing further the constraints the system stays isostatic up to the point where the volume of available phase space shrinks to zero. We derive our results using the replica method and we also analyze a dynamical algorithm, the Karush-Kuhn-Tucker algorithm, through dynamical mean-field theory and we show how to recover the results of the replica approach in the replica symmetric phase.


  • Open Access

    Proliferation of non-linear excitations in the piecewise-linear perceptron

    by Antonio Sclocchi, Pierfrancesco Urbani

    We investigate the properties of local minima of the energy landscape of a continuous non-convex optimization problem, the spherical perceptron with piecewise linear cost function and show that they are critical, marginally stable and displaying a set of pseudogaps, singularities and non-linear excitations whose properties appear to be in the same universality class of jammed packings of hard spheres. The piecewise linear perceptron problem appears as an evolution of the purely linear perceptron optimization problem that has been recently investigated in [1]. Its cost function contains two non-analytic points where the derivative has a jump. Correspondingly, in the non-convex/glassy phase, these two points give rise to four pseudogaps in the force distribution and this induces four power laws in the gap distribution as well. In addition one can define an extended notion of isostaticity and show that local minima appear again to be isostatic in this phase. We believe that our results generalize naturally to more complex cases with a proliferation of non-linear excitations as the number of non-analytic points in the cost function is increased.


  • Open Access

    Critical energy landscape of linear soft spheres

    by Silvio Franz, Antonio Sclocchi, Pierfrancesco Urbani

    We show that soft spheres interacting with a linear ramp potential when overcompressed beyond the jamming point fall in an amorphous solid phase which is critical, mechanically marginally stable and share many features with the jamming point itself. In the whole phase, the relevant local minima of the potential energy landscape display an isostatic contact network of perfectly touching spheres whose statistics is controlled by an infinite lengthscale. Excitations around such energy minima are non-linear, system spanning, and characterized by a set of non-trivial critical exponents. We perform numerical simulations to measure their values and show that, while they coincide, within numerical precision, with the critical exponents appearing at jamming, the nature of the corresponding excitations is richer. Therefore, linear soft spheres appear as a novel class of finite dimensional systems that self-organize into new, critical, marginally stable, states.