Reproducibility and Discriminability of Brain Patterns of Semantic Categories Enhanced by Congruent Audiovisual Stimuli
Open Access
- 29 June 2011
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 6 (6) , e20801
Abstract
One of the central questions in cognitive neuroscience is the precise neural representation, or brain pattern, associated with a semantic category. In this study, we explored the influence of audiovisual stimuli on the brain patterns of concepts or semantic categories through a functional magnetic resonance imaging (fMRI) experiment. We used a pattern search method to extract brain patterns corresponding to two semantic categories: “old people” and “young people.” These brain patterns were elicited by semantically congruent audiovisual, semantically incongruent audiovisual, unimodal visual, and unimodal auditory stimuli belonging to the two semantic categories. We calculated the reproducibility index, which measures the similarity of the patterns within the same category. We also decoded the semantic categories from these brain patterns. The decoding accuracy reflects the discriminability of the brain patterns between two categories. The results showed that both the reproducibility index of brain patterns and the decoding accuracy were significantly higher for semantically congruent audiovisual stimuli than for unimodal visual and unimodal auditory stimuli, while the semantically incongruent stimuli did not elicit brain patterns with significantly higher reproducibility index or decoding accuracy. Thus, the semantically congruent audiovisual stimuli enhanced the within-class reproducibility of brain patterns and the between-class discriminability of brain patterns, and facilitate neural representations of semantic categories or concepts. Furthermore, we analyzed the brain activity in superior temporal sulcus and middle temporal gyrus (STS/MTG). The strength of the fMRI signal and the reproducibility index were enhanced by the semantically congruent audiovisual stimuli. Our results support the use of the reproducibility index as a potential tool to supplement the fMRI signal amplitude for evaluating multimodal integration.Keywords
This publication has 36 references indexed in Scilit:
- The confounding effect of response amplitude on MVPA performance measuresNeuroImage, 2011
- Greater Neural Pattern Similarity Across Repetitions Is Associated with Better MemoryScience, 2010
- Machine learning classifiers and fMRI: A tutorial overviewNeuroImage, 2009
- Decoding reveals the contents of visual working memory in early visual areasNature, 2009
- "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and SpeechScience, 2008
- Identifying natural images from human brain activityNature, 2008
- Inverse retinotopy: Inferring the visual content of images from brain activation patternsNeuroImage, 2006
- Binding crossmodal object features in perirhinal cortexProceedings of the National Academy of Sciences, 2006
- On Representation and ReproducibilityJournal of Cognitive Neuroscience, 2003
- Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortexNeuroImage, 2003