A large-scale examination of inductive biases shaping high-level visual representation in brains and machines
Top Cited Papers
Open Access
- 30 October 2024
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 15 (1) , 1-18
Abstract
The rapid release of high-performing computer vision models offers new potential to study the impact of different inductive biases on the emergent brain alignment of learned representations. Here, we perform controlled comparisons among a curated set of 224 diverse models to test the impact of specific model properties on visual brain predictivity – a process requiring over 1.8 billion regressions and 50.3 thousand representational similarity analyses. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and task objectives (e.g. purely visual contrastive learning versus vision- language alignment) achieve near equivalent brain predictivity, when other factors are held constant. Instead, variation across visual training diets yields the largest, most consistent effect on brain predictivity. Many models achieve similarly high brain predictivity, despite clear variation in their underlying representations – suggesting that standard methods used to link models to brains may be too flexible. Broadly, these findings challenge common assumptions about the factors underlying emergent brain alignment, and outline how we can leverage controlled model comparison to probe the common computational principles underlying biological and artificial visual systems.Keywords
Funding Information
- National Science Foundation (NSF-CAREER BCS-1942438)
This publication has 90 references indexed in Scilit:
- Explanatory models in neuroscience, Part 1: Taking mechanistic abstraction seriouslyCognitive Systems Research, 2024
- SLIP: Self-supervision Meets Language-Image Pre-training2022
- Brain hierarchy score: Which deep neural networks are hierarchically brain-like?iScience, 2021
- Extensive sampling for complete models of individual brainsCurrent Opinion in Behavioral Sciences, 2021
- Deep Randomized Neural Networks2020
- Seeing it all: Convolutional network layers map the function of the human visual systemNeuroImage, 2017
- Fixed versus mixed RSA: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational modelsJournal of Mathematical Psychology, 2017
- Microsoft COCO: Common Objects in Context2014
- A comprehensive assessment of the structural similarity indexSignal, Image and Video Processing, 2009
- Estimating regression models with unknown break‐pointsStatistics in Medicine, 2003