Jacobian-Based Interpretation of Nonlinear Neural Encoding Model
Preprint
Abstract
In recent years, the alignment between artificial neural network (ANN) embeddings and blood oxygenation level dependent (BOLD) responses in functional magnetic resonance imaging (fMRI) via neural encoding models has significantly advanced research on neural representation mechanisms and interpretability in the brain. However, these approaches remain limited in characterizing the brain's inherently nonlinear response properties. To address this, we propose the Jacobian-based Nonlinearity Evaluation (JNE), an interpretability metric for nonlinear neural encoding models. JNE quantifies nonlinearity by statistically measuring the dispersion of local linear mappings (Jacobians) from model representations to predicted BOLD responses, thereby approximating the nonlinearity of BOLD signals. Centered on proposing JNE as a novel interpretability metric, we validated its effectiveness through controlled simulation experiments on various activation functions and network architectures, and further verified it on real fMRI data, demonstrating a hierarchical progression of nonlinear characteristics from primary to higher-order visual cortices, consistent with established cortical organization. We further extended JNE with Sample-Specificity (JNE-SS), revealing stimulus-selective nonlinear response patterns in functionally specialized brain regions. As the first interpretability metric for quantifying nonlinear responses, JNE provides new insights into brain information processing. Code available at https://github.com/Gaitxh/JNE.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Distributed representations of behaviour-derived object dimensions in the human visual systemNature Human Behaviour, 2024
- Shared functional specialization in transformer-based language models and the human brainNature Communications, 2024
- A novel residual-based Bayesian expectation–maximization adaptive Kalman filter with inaccurate and time-varying noise covariancesMeasurement, 2024
- Enhancing neural encoding models for naturalistic perception with a multi-level integration of deep neural networks and cortical networksScience Bulletin, 2024
- Shared computational principles for language processing in humans and deep language modelsNature Neuroscience, 2022
- An awareness-dependent mapping of saliency in the human visual systemNeuroImage, 2022
- A visual encoding model based on deep neural networks and transfer learning for brain activity measured by functional magnetic resonance imagingJournal of Neuroscience Methods, 2019
- Nonlinear Dynamics in Speech Perception2010
- Investigating the source of BOLD nonlinearity in human visual cortex in response to paired visual stimuliNeuroImage, 2008
- The neural basis of the hemodynamic response nonlinearity in human primary visual cortex: Implications for neurovascular coupling mechanismNeuroImage, 2006