Leonardo Petrini
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Leonardo Petrini has a total of 7 co-authors. Here are the top co-authors by number of shared publications:
Matthieu Wyart
Institute of Physics, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
2 shared publications- Umberto Maria Tomasini
EPFL
2 shared publications - Alessandro Favero
École Polytechnique Fédérale de Lausanne (EPFL)
1 shared publication Francesco Cagnetta
Institute of Physics, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
1 shared publication- Francesco Cagnetta
Ecole Polytech Fed Lausanne EPFL, Inst Phys, Lausanne, Switzerland
1 shared publication - Mario Geiger
NVIDIA, Santa Clara, USA
1 shared publication
Latest Publications
- Open Access
- 1 July 2024
- Published by American Physical Society (APS) in Physical Review X
- Vol. 14 (3) , 031001
How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
Deep learning algorithms demonstrate a surprising ability to learn high-dimensional tasks from limited examples. This is commonly attributed to the depth of neural networks, enabling them to build a hierarchy of abstract, low-dimensional data representations. However, how many training examples are required to learn such representations remains unknown. To quantitatively study this question, we introduce the random hierarchy model: a family of synthetic tasks inspired by the hierarchical structure of language and images. The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class. In turn, each feature corresponds to a group of subfeatures chosen among several equivalent groups and so on, following a hierarchy of composition rules. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Moreover, the number of data required corresponds to the point where correlations between low-level features and classes become detectable. Overall, our results indicate how deep networks overcome the curse of dimensionality by building invariant representations and provide an estimate of the number of data required to learn a hierarchical task. Published by the American Physical Society 2024
- 1 July 2024
- Published by American Physical Society (APS) in Physical Review X
- Vol. 14 (3) , 031001
Open AccessHow Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model
Deep learning algorithms demonstrate a surprising ability to learn high-dimensional tasks from limited examples. This is commonly attributed to the depth of neural networks, enabling them to build a hierarchy of abstract, low-dimensional data representations. However, how many training examples are required to learn such representations remains unknown. To quantitatively study this question, we introduce the random hierarchy model: a family of synthetic tasks inspired by the hierarchical structure of language and images. The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class. In turn, each feature corresponds to a group of subfeatures chosen among several equivalent groups and so on, following a hierarchy of composition rules. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Moreover, the number of data required corresponds to the point where correlations between low-level features and classes become detectable. Overall, our results indicate how deep networks overcome the curse of dimensionality by building invariant representations and provide an estimate of the number of data required to learn a hierarchical task. Published by the American Physical Society 2024
- Open Access
- 9 November 2023
- Published by IOP Publishing in Machine Learning: Science and Technology
- Vol. 4 (4) , 045026
How deep convolutional neural networks lose spatial information with training
A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image datasets, this view is supported by the observation that after (and not before) training, the neural representation becomes less and less sensitive to diffeomorphisms acting on images as the signal propagates through the network. This loss of sensitivity correlates with performance and surprisingly correlates with a gain of sensitivity to white noise acquired during training. Which are the mechanisms learned by convolutional neural networks (CNNs) responsible for the these phenomena? In particular, why is the sensitivity to noise heightened with training? Our approach consists of two steps. (1) Analyzing the layer-wise representations of trained CNNs, we disentangle the role of spatial pooling in contrast to channel pooling in decreasing their sensitivity to image diffeomorphisms while increasing their sensitivity to noise. (2) We introduce model scale-detection tasks, which qualitatively reproduce the phenomena reported in our empirical analysis. In these models we can assess quantitatively how spatial pooling affects these sensitivities. We find that the increased sensitivity to noise observed in deep ReLU networks is a mechanistic consequence of the perturbing noise piling up during spatial pooling, after being rectified by ReLU units. Using odd activation functions like tanh drastically reduces the CNNs’ sensitivity to noise.
- 9 November 2023
- Published by IOP Publishing in Machine Learning: Science and Technology
- Vol. 4 (4) , 045026
Open AccessHow deep convolutional neural networks lose spatial information with training
A central question of machine learning is how deep nets manage to learn tasks in high dimensions. An appealing hypothesis is that they achieve this feat by building a representation of the data where information irrelevant to the task is lost. For image datasets, this view is supported by the observation that after (and not before) training, the neural representation becomes less and less sensitive to diffeomorphisms acting on images as the signal propagates through the network. This loss of sensitivity correlates with performance and surprisingly correlates with a gain of sensitivity to white noise acquired during training. Which are the mechanisms learned by convolutional neural networks (CNNs) responsible for the these phenomena? In particular, why is the sensitivity to noise heightened with training? Our approach consists of two steps. (1) Analyzing the layer-wise representations of trained CNNs, we disentangle the role of spatial pooling in contrast to channel pooling in decreasing their sensitivity to image diffeomorphisms while increasing their sensitivity to noise. (2) We introduce model scale-detection tasks, which qualitatively reproduce the phenomena reported in our empirical analysis. In these models we can assess quantitatively how spatial pooling affects these sensitivities. We find that the increased sensitivity to noise observed in deep ReLU networks is a mechanistic consequence of the perturbing noise piling up during spatial pooling, after being rectified by ReLU units. Using odd activation functions like tanh drastically reduces the CNNs’ sensitivity to noise.
- Open Access
- 1 August 2021
- Published by Elsevier in Physics Reports
- Vol. 924, 1-18
Landscape and training regimes in deep learning
- 1 August 2021
- Published by Elsevier in Physics Reports
- Vol. 924, 1-18
Open AccessLandscape and training regimes in deep learning