Cross-modal representations in early visual and auditory cortices revealed by multi-voxel pattern analysis
- 10 June 2019
- journal article
- research article
- Published by Springer Nature in Brain Imaging and Behavior
- Vol. 14 (5) , 1908-1920
Abstract
Primary sensory cortices can respond not only to their defined sensory modality but also to cross-modal information. In addition to the observed cross-modal phenomenon, it is valuable to research further whether cross-modal information can be valuable for categorizing stimuli and what effect other factors, such as experience and imagination, may have on cross-modal processing. In this study, we researched cross-modal information processing in the early visual cortex (EVC, including the visual area 1, 2, and 3 (V1, V2, and V3)) and auditory cortex (primary (A1) and secondary (A2) auditory cortex). Images and sound clips were presented to participants separately in two experiments in which participants’ imagination and expectations were restricted by an orthogonal fixation task and the data were collected by functional magnetic resonance imaging (fMRI). We successfully decoded categories of the cross-modal stimuli in the ROIs except for V1 by multi-voxel pattern analysis (MVPA). It was further shown that familiar sounds had the advantage of classification accuracies in V2 and V3 when compared with unfamiliar sounds. The results of the cross-classification analysis showed that there was no significant similarity between the activity patterns induced by different stimulus modalities. Even though the cross-modal representation is robust when considering the restriction of top-down expectations and mental imagery in our experiments, the sound experience showed effects on cross-modal representation in V2 and V3. In addition, primary sensory cortices may receive information from different modalities in different ways, so the activity patterns between two modalities were not similar enough to complete the cross-classification successfully.Keywords
Funding Information
- National Natural Science Foundation of China (U1736219, 61571327)
This publication has 51 references indexed in Scilit:
- Auditory modulation of visual stimulus encoding in human retinotopic cortexNeuroImage, 2013
- Combined diffusion-weighted and functional magnetic resonance imaging reveals a temporal-occipital network involved in auditory-visual object processingFrontiers in Integrative Neuroscience, 2013
- Sight and Sound Converge to Form Modality-Invariant Representations in Temporoparietal CortexThe Journal of Neuroscience, 2012
- Sound-Driven Synaptic Inhibition in Primary Visual CortexNeuron, 2012
- Current perspectives and methods in studying neural mechanisms of multisensory interactionsNeuroscience & Biobehavioral Reviews, 2012
- Seeing Touch Is Correlated with Content-Specific Activity in Primary Somatosensory CortexCerebral Cortex, 2011
- Predicting visual stimuli on the basis of activity in auditory corticesNature Neuroscience, 2010
- Decoding reveals the contents of visual working memory in early visual areasNature, 2009
- A cross‐modal system linking primary auditory and visual cortices: Evidence from intrinsic fMRI connectivity analysisHuman Brain Mapping, 2008
- Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and JudgmentsNeuron, 2008