Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech
Preprint
Abstract
Natural language contains information at multiple timescales. To understand how the human brain represents this information, one approach is to build encoding models that predict fMRI responses to natural language using representations extracted from neural network language models (LMs). However, these LM-derived representations do not explicitly separate information at different timescales, making it difficult to interpret the encoding models. In this work we construct interpretable multi-timescale representations by forcing individual units in an LSTM LM to integrate information over specific temporal scales. This allows us to explicitly and directly map the timescale of information encoded by each individual fMRI voxel. Further, the standard fMRI encoding procedure does not account for varying temporal properties in the encoding features. We modify the procedure so that it can capture both short- and long-timescale information. This approach outperforms other encoding models, particularly for voxels that represent long-timescale information. It also provides a finer-grained map of timescale information in the human language pathway. This serves as a framework for future work investigating temporal hierarchies across artificial and biological language systems.Keywords
This publication has 21 references indexed in Scilit:
- A hierarchy of intrinsic timescales across primate cortexNature Neuroscience, 2014
- A Selected Core Microbiome Drives the Early Stages of Three Popular Italian Cheese ManufacturesPLOS ONE, 2014
- A place for time: the spatiotemporal structure of neural dynamics during natural auditionJournal of Neurophysiology, 2013
- Slow Cortical Dynamics and the Accumulation of Information over Long TimescalesNeuron, 2012
- Reconstructing Visual Experiences from Brain Activity Evoked by Natural MoviesCurrent Biology, 2011
- Topographic Mapping of a Hierarchy of Temporal Receptive Windows Using a Narrated StoryThe Journal of Neuroscience, 2011
- The Temporal Structures and Functional Significance of Scale-free Brain ActivityNeuron, 2010
- Predicting Human Brain Activity Associated with the Meanings of NounsScience, 2008
- Speaker identification on the SCOTUS corpusThe Journal of the Acoustical Society of America, 2008
- COMPLETE FUNCTIONAL CHARACTERIZATION OF SENSORY NEURONS BY SYSTEM IDENTIFICATIONAnnual Review of Neuroscience, 2006