Multimodal deep neural decoding of visual object representation in humans
Preprint
Abstract
Perception and categorization of objects in a visual scene are essential to grasp the surrounding situation. However, it is unclear how neural activities in spatially distributed brain regions, especially in terms of temporal dynamics, represent visual objects. To address this issue, we explored the spatial and temporal organization of visual object representations using concurrent functional magnetic resonance imaging (fMRI) and electroencephalography (EEG), combined with neural decoding using deep neural networks (DNNs). Visualization of the fMRI DNN revealed that visual categorization (faces or non-face objects) occurred in brain-wide cortical regions, including the ventral temporal cortex. Interestingly, the EEG DNN valued the earlier phase of neural responses for categorization and the later phase of neural responses for sub-categorization. Combination of the two DNNs improved the classification performance for both categorization and sub-categorization. These deep learning-based results demonstrate a categorization principle in which visual objects are represented in a spatially organized and coarse-to-fine manner.Keywords
This publication has 74 references indexed in Scilit:
- Category-selective phase coding in the superior temporal sulcusProceedings of the National Academy of Sciences, 2012
- Large-scale automated synthesis of human functional neuroimaging dataNature Methods, 2011
- Unraveling the distributed neural code of facial identity through spatiotemporal pattern analysisProceedings of the National Academy of Sciences, 2011
- The Timing of Visual Object CategorizationFrontiers in Psychology, 2011
- Differential Contribution of Right and Left Temporo-Occipital and Anterior Temporal Lesions to Face Recognition DisordersFrontiers in Human Neuroscience, 2011
- Ensemble-based classifiersArtificial Intelligence Review, 2009
- Matching Categorical Object Representations in Inferior Temporal Cortex of Man and MonkeyNeuron, 2008
- Individual faces elicit distinct response patterns in human anterior temporal cortexProceedings of the National Academy of Sciences, 2007
- A feedforward architecture accounts for rapid categorizationProceedings of the National Academy of Sciences, 2007
- Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal CortexScience, 2001