Cross-Modal Interaction Between Auditory and Visual Input Impacts Memory Retrieval
Authors:
Open Access
- 26 July 2021
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Neuroscience
- Vol. 15
Abstract
How we perceive and learn about our environment is influenced by our prior experiences and existing representations of the world. Top-down cognitive processes, such as attention and expectations, can alter how we process sensory stimuli, both within a modality (e.g., effects of auditory experience on auditory perception), as well as across modalities (e.g., effects of visual feedback on sound localization). Here, we demonstrate that experience with different types of auditory input (spoken words vs. environmental sounds) modulates how humans remember concurrently-presented visual objects. Participants viewed a series of line drawings (e.g., picture of a cat) displayed in one of four quadrants while listening to a word or sound that was congruent (e.g., “cat” or ), incongruent (e.g., “motorcycle” or ), or neutral (e.g., a meaningless pseudoword or a tonal beep) relative to the picture. Following the encoding phase, participants were presented with the original drawings plus new drawings and asked to indicate whether each one was “old” or “new.” If a drawing was designated as “old,” participants then reported where it had been displayed. We find that words and sounds both elicit more accurate memory for what objects were previously seen, but only congruent environmental sounds enhance memory for where objects were positioned – this, despite the fact that the auditory stimuli were not meaningful spatial cues of the objects’ locations on the screen. Given that during real-world listening conditions, environmental sounds, but not words, reliably originate from the location of their referents, listening to sounds may attune the visual dorsal pathway to facilitate attention and memory for objects’ locations. We propose that audio-visual associations in the environment and in our previous experience jointly contribute to visual memory, strengthening visual memory through exposure to auditory input.Keywords
Funding Information
- Eunice Kennedy Shriver National Institute of Child Health and Human Development (R01HD059858)
This publication has 123 references indexed in Scilit:
- Attentional capture of objects referred to by spoken language.Journal of Experimental Psychology: Human Perception and Performance, 2011
- Object-based auditory facilitation of visual search for pictures and words with frequent and rare targetsActa Psychologica, 2011
- The incongruency advantage for environmental sounds presented in natural auditory scenes.Journal of Experimental Psychology: Human Perception and Performance, 2011
- The multifaceted interplay between attention and multisensory integrationTrends in Cognitive Sciences, 2010
- Conceptual distinctiveness supports detailed visual long-term memory for real-world objects.Journal of Experimental Psychology: General, 2010
- Early word-learning entails reference, not merely associationsTrends in Cognitive Sciences, 2009
- Spatiotemporal architecture of cortical receptive fields and its impact on multisensory interactionsExperimental Brain Research, 2009
- Multisensory Interplay Reveals Crossmodal Influences on ‘Sensory-Specific’ Brain Regions, Neural Responses, and JudgmentsNeuron, 2008
- Recognition memory: What are the roles of the perirhinal cortex and hippocampus?Nature Reviews Neuroscience, 2001
- Hearing lips and seeing voicesNature, 1976