Convergent and invariant object representations for sight, sound, and touch
Authors:
- 5 June 2015
- journal article
- Published by Wiley in Human Brain Mapping
- Vol. 36 (9) , 3629-3640
Abstract
We continuously perceive objects in the world through multiple sensory channels. In this study, we investigated the convergence of information from different sensory streams within the cerebral cortex. We presented volunteers with three common objects via three different modalities—sight, sound, and touch—and used multivariate pattern analysis of functional magnetic resonance imaging data to map the cortical regions containing information about the identity of the objects. We could reliably predict which of the three stimuli a subject had seen, heard, or touched from the pattern of neural activity in the corresponding early sensory cortices. Intramodal classification was also successful in large portions of the cerebral cortex beyond the primary areas, with multiple regions showing convergence of information from two or all three modalities. Using crossmodal classification, we also searched for brain regions that would represent objects in a similar fashion across different modalities of presentation. We trained a classifier to distinguish objects presented in one modality and then tested it on the same objects presented in a different modality. We detected audiovisual invariance in the right temporo‐occipital junction, audiotactile invariance in the left postcentral gyrus and parietal operculum, and visuotactile invariance in the right postcentral and supramarginal gyri. Our maps of multisensory convergence and crossmodal generalization reveal the underlying organization of the association cortices, and may be related to the neural basis for mental concepts. Hum Brain Mapp 36:3629–3640, 2015. © 2015 Wiley Periodicals, Inc.Keywords
Funding Information
- The Mathers Foundation
- the National Institutes of Health (NIH/NINDS 5P50NS019632.)
This publication has 49 references indexed in Scilit:
- Heteromodal conceptual processing in the angular gyrusNeuroImage, 2013
- Decoding semantics across fMRI sessions with different stimulus modalities: a practical MVPA studyFrontiers in Neuroinformatics, 2012
- Semantics and the multisensory brain: How meaning modulates processes of audio-visual integrationBrain Research, 2008
- Functional imaging of human crossmodal identification and object recognitionExperimental Brain Research, 2005
- Statistical Criteria in fMRI Studies of Multisensory IntegrationNeuroinformatics, 2005
- Polymodal Motion Processing in Posterior Parietal and Premotor CortexNeuron, 2001
- The Psychophysics ToolboxSpatial Vision, 1997
- Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognitionCognition, 1989
- The Brain Binds Entities and Events by Multiregional Activation from Convergence ZonesNeural Computation, 1989
- 3. Concepts in the BrainMind & Language, 1989