A Spectral Theory of Neural Prediction and Alignment
Preprint
Abstract
The representations of neural networks are often compared to those of biological systems by performing regression between the neural network responses and those measured from biological systems. Many different state-of-the-art deep neural networks yield similar neural predictions, but it remains unclear how to differentiate among models that perform equally well at predicting neural responses. To gain insight into this, we use a recent theoretical framework that relates the generalization error from regression to the spectral properties of the model and the target. We apply this theory to the case of regression between model activations and neural responses and decompose the neural prediction error in terms of the model eigenspectra, alignment of model eigenvectors and neural responses, and the training set size. Using this decomposition, we introduce geometrical measures to interpret the neural prediction error. We test a large number of deep neural networks that predict visual cortical activity and show that there are multiple types of geometries that result in low neural prediction error as measured via regression. The work demonstrates that carefully decomposing representational metrics can provide interpretability of how models are capturing neural activity and points the way towards improved models of neural activity.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- A self-supervised domain-general learning framework for human ventral stream representationNature Communications, 2022
- Neural population geometry: An approach for understanding biological and artificial neural networksCurrent Opinion in Neurobiology, 2021
- Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activityCurrent Opinion in Neurobiology, 2021
- Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networksNature Communications, 2021
- If deep learning is the answer, what is the question?Nature Reviews Neuroscience, 2020
- Separability and geometry of object manifolds in deep neural networksNature Communications, 2020
- High-dimensional geometry of population responses in visual cortexNature, 2019
- A functional and perceptual signature of the second visual area in primatesNature Neuroscience, 2013
- Spectral Analysis of Large Dimensional Random Matrices2010
- A back-propagation programmed network that simulates response properties of a subset of posterior parietal neuronsNature, 1988