Mechanism for feature learning in neural networks and backpropagation-free machine learning models
Top Cited Papers
- 29 March 2024
- journal article
- research article
- Published by American Association for the Advancement of Science (AAAS) in Science
- Vol. 383 (6690) , 1461-1467
Abstract
Understanding how neural networks learn features, or relevant patterns in data, for prediction is necessary for their reliable use in technological and scientific applications. In this work, we presented a unifying mathematical mechanism, known as average gradient outer product (AGOP), that characterized feature learning in neural networks. We provided empirical evidence that AGOP captured features learned by various neural network architectures, including transformer-based language models, convolutional networks, multilayer perceptrons, and recurrent neural networks. Moreover, we demonstrated that AGOP, which is backpropagation-free, enabled feature learning in machine learning models, such as kernel machines, that a priori could not identify task-specific features. Overall, we established a fundamental mechanism that captured feature learning in neural networks and enabled feature learning in general machine learning models.Keywords
This publication has 24 references indexed in Scilit:
- Highly accurate protein structure prediction for the human proteomeNature, 2021
- Wide neural networks of any depth evolve as linear models under gradient descent *Journal of Statistical Mechanics: Theory and Experiment, 2020
- Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma RecognitionJAMA Dermatology, 2019
- Rotation Equivariant CNNs for Digital Pathology2018
- Deep Residual Learning for Image Recognition2016
- Deep Learning Face Attributes in the Wild2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Describing Textures in the Wild2014
- Visualizing and Understanding Convolutional Networks2014
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998