A Theory of Neural Tangent Kernel Alignment and Its Influence on Training
Preprint
Abstract
The training dynamics and generalization properties of neural networks (NN) can be precisely characterized in function space via the neural tangent kernel (NTK). Structural changes to the NTK during training reflect feature learning and underlie the superior performance of networks outside of the static kernel regime. In this work, we seek to theoretically understand kernel alignment, a prominent and ubiquitous structural change that aligns the NTK with the target function. We first study a toy model of kernel evolution in which the NTK evolves to accelerate training and show that alignment naturally emerges from this demand. We then study alignment mechanism in deep linear networks and two layer ReLU networks. These theories provide good qualitative descriptions of kernel alignment and specialization in practical networks and identify factors in network architecture and data structure that drive kernel alignment. In nonlinear networks with multiple outputs, we identify the phenomenon of kernel specialization, where the kernel function for each output head preferentially aligns to its own target function. Together, our results provide a mechanistic explanation of how kernel alignment emerges during NN training and a normative explanation of how it benefits training.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Explaining neural scaling lawsProceedings of the National Academy of Sciences, 2024
- A rigorous framework for the mean field limit of multilayer neural networksMathematical Statistics and Learning, 2023
- The Principles of Deep Learning Theory2022
- When do neural networks outperform kernel methods?*Journal of Statistical Mechanics: Theory and Experiment, 2021
- Neural tangent kernel: convergence and generalization in neural networks (invited paper)2021
- Geometric compression of invariant manifolds in neural networksJournal of Statistical Mechanics: Theory and Experiment, 2021
- Wide neural networks of any depth evolve as linear models under gradient descent *Journal of Statistical Mechanics: Theory and Experiment, 2020
- High-dimensional dynamics of generalization error in neural networksNeural Networks, 2020
- Disentangling feature and lazy training in deep neural networksJournal of Statistical Mechanics: Theory and Experiment, 2020
- Mean Field Analysis of Neural Networks: A Law of Large NumbersSIAM Journal on Applied Mathematics, 2020