Sight and Sound Converge to Form Modality-Invariant Representations in Temporoparietal Cortex
Authors:
Open Access
- 21 November 2012
- journal article
- Published by Society for Neuroscience in The Journal of Neuroscience
- Vol. 32 (47) , 16629-16636
Abstract
People can identify objects in the environment with remarkable accuracy, regardless of the sensory modality they use to perceive them. This suggests that information from different sensory channels converges somewhere in the brain to form modality-invariant representations, i.e., representations that reflect an object independently of the modality through which it has been apprehended. In this functional magnetic resonance imaging study of human subjects, we first identified brain areas that responded to both visual and auditory stimuli and then used crossmodal multivariate pattern analysis to evaluate the neural representations in these regions for content specificity (i.e., do different objects evoke different representations?) and modality invariance (i.e., do the sight and the sound of the same object evoke a similar representation?). While several areas became activated in response to both auditory and visual stimulation, only the neural patterns recorded in a region around the posterior part of the superior temporal sulcus displayed both content specificity and modality invariance. This region thus appears to play an important role in our ability to recognize objects in our surroundings through multiple sensory channels and to process them at a supramodal (i.e., conceptual) level.Keywords
This publication has 42 references indexed in Scilit:
- Disentangling visual imagery and perception of real-world objectsNeuroImage, 2012
- Information mapping with pattern classifiers: A comparative studyNeuroImage, 2011
- LIBSVMACM Transactions on Intelligent Systems and Technology, 2011
- Seeing Touch Is Correlated with Content-Specific Activity in Primary Somatosensory CortexCerebral Cortex, 2011
- Commonality of neural representations of words and picturesNeuroImage, 2011
- Predicting visual stimuli on the basis of activity in auditory corticesNature Neuroscience, 2010
- Reading the mind's eye: Decoding category information during mental imageryNeuroImage, 2010
- PyMVPA: a Python Toolbox for Multivariate Pattern Analysis of fMRI DataNeuroinformatics, 2009
- Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and JudgmentsNeuron, 2008
- Mental Concerts: Musical Imagery and Auditory CortexNeuron, 2005