Group‐level brain decoding with deep learning
Authors:
Open Access
- 27 September 2023
- journal article
- research article
- Published by Wiley in Human Brain Mapping
- Vol. 44 (17) , 6105-6119
Abstract
Decoding brain imaging data are gaining popularity, with applications in brain‐computer interfaces and the study of neural representations. Decoding is typically subject‐specific and does not generalise well over subjects, due to high amounts of between subject variability. Techniques that overcome this will not only provide richer neuroscientific insights but also make it possible for group‐level models to outperform subject‐specific models. Here, we propose a method that uses subject embedding, analogous to word embedding in natural language processing, to learn and exploit the structure in between‐subject variability as part of a decoding model, our adaptation of the WaveNet architecture for classification. We apply this to magnetoencephalography data, where 15 subjects viewed 118 different images, with 30 examples per image; to classify images using the entire 1 s window following image presentation. We show that the combination of deep learning and subject embedding is crucial to closing the performance gap between subject‐ and group‐level decoding models. Importantly, group models outperform subject models on low‐accuracy subjects (although slightly impair high‐accuracy subjects) and can be helpful for initialising subject models. While we have not generally found group‐level models to perform better than subject‐level models, the performance of group modelling is expected to be even higher with bigger datasets. In order to provide physiological interpretation at the group level, we make use of permutation feature importance. This provides insights into the spatiotemporal and spectral information encoded in the models. All code is available on GitHub (https://github.com/ricsinaruto/MEG-group-decode).Keywords
All Related Versions
Funding Information
- Wellcome Trust (203139/Z/16/Z, 215573/Z/19/Z, 106183/Z/14/Z)
This publication has 42 references indexed in Scilit:
- Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondenceScientific Reports, 2016
- Deep Neural Networks: A New Framework for Modeling Biological Vision and Brain Information ProcessingAnnual Review of Vision Science, 2015
- MEG decoding across subjects2014
- Direct classification of all American English phonemes using signals from functional speech motor cortexJournal of Neural Engineering, 2014
- MEG and EEG data analysis with MNE-PythonFrontiers in Neuroscience, 2013
- Permutation importance: a corrected feature importance measureBioinformatics, 2010
- Identifying natural images from human brain activityNature, 2008
- Matplotlib: A 2D Graphics EnvironmentComputing in Science & Engineering, 2007
- Distinguishing low frequency oscillations within the 1/f spectral behaviour of electromagnetic brain signalsBehavioral and Brain Functions, 2007
- Electrophysiological evidence for notation independence in numerical processingBehavioral and Brain Functions, 2007