Universality of Winning Tickets: A Renormalization Group Perspective
Preprint
Abstract
Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. This has generated broad interest, but methods to study this universality are lacking. We make use of renormalization group theory, a powerful tool from theoretical physics, to address this need. We find that iterative magnitude pruning, the principal algorithm used for discovering winning tickets, is a renormalization group scheme, and can be viewed as inducing a flow in parameter space. We demonstrate that ResNet-50 models with transferable winning tickets have flows with common properties, as would be expected from the theory. Similar observations are made for BERT models, with evidence that their flows are near fixed points. Additionally, we leverage our framework to study winning tickets transferred across ResNet architectures, observing that smaller models have flows with more uniform properties than larger models, complicating transfer between them.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- The Principles of Deep Learning Theory2022
- Mutual information, neural networks and the renormalization groupNature Physics, 2018
- Why Does Deep and Cheap Learning Work So Well?Journal of Statistical Physics, 2017
- PCA Meets RGJournal of Statistical Physics, 2017
- Rheology of Polymers Near Liquid-Solid Transitions2007
- Spectral Properties of Dynamical Systems, Model Reduction and DecompositionsNonlinear Dynamics, 2005
- Renormalization group analysis of the small-world network modelPhysics Letters A, 1999
- Universality Classes of Inelastic Electron Scattering Cross-sectionsSurface and Interface Analysis, 1997
- Quasiperiodicity in dissipative systems: A renormalization group analysisPhysica D: Nonlinear Phenomena, 1982
- Existence of a phase-transition in a one-dimensional Ising ferromagnetCommunications in Mathematical Physics, 1969