Facemap: a framework for modeling neural activity based on orofacial tracking
Preprint
Abstract
Recent studies in mice have shown that orofacial behaviors drive a large fraction of neural activity across the brain. To understand the nature and function of these signals, we need better computational models to characterize the behaviors and relate them to neural activity. Here we developed Facemap, a framework consisting of a keypoint tracking algorithm and a deep neural network encoder for predicting neural activity. We used the Facemap keypoints as input for the deep neural network to predict the activity of ∼50,000 simultaneously-recorded neurons and in visual cortex we doubled the amount of explained variance compared to previous methods. Our keypoint tracking algorithm was more accurate than existing pose estimation tools, while the inference speed was several times faster, making it a powerful tool for closed-loop behavioral experiments. The Facemap tracker was easy to adapt to data from new labs, requiring as few as 10 annotated frames for near-optimal performance. We used Facemap to find that the neuronal activity clusters which were highly driven by behaviors were more spatially spread-out across cortex. We also found that the deep keypoint features inferred by the model had time-asymmetrical state dynamics that were not apparent in the raw keypoint data. In summary, Facemap provides a stepping stone towards understanding the function of the brainwide neural signals and their relation to behavior.Keywords
All Related Versions
- Published version: Nature Neuroscience, 27 (1), 187.
This publication has 58 references indexed in Scilit:
- High-precision coding in visual cortexCell, 2021
- The Emergence of the Spatial Structure of Tectal Spontaneous Activity Is Independent of Visual InputsCell Reports, 2017
- DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model2016
- Mapping Sub-Second Structure in Mouse BehaviorNeuron, 2015
- Locomotion and Task Demands Differentially Modulate Thalamic Audiovisual Processing during Active SearchCurrent Biology, 2015
- Arousal and Locomotion Make Distinct Contributions to Cortical Activity Patterns and Visual EncodingNeuron, 2015
- Effects of Locomotion Extend throughout the Mouse Early Visual SystemCurrent Biology, 2014
- Modulation of Visual Responses by Behavioral State in Mouse Visual CortexNeuron, 2010
- Spontaneous and driven cortical activity: implications for computationCurrent Opinion in Neurobiology, 2009
- Reduced-rank regression for the multivariate linear modelJournal of Multivariate Analysis, 1975