Images with harder-to-reconstruct visual representations leave stronger memory traces
- 13 May 2024
- journal article
- research article
- Published by Springer Nature in Nature Human Behaviour
- Vol. 8 (7) , 1309-1320
Abstract
Much of what we remember is not because of intentional selection, but simply a by-product of perceiving. This raises a foundational question about the architecture of the mind: how does perception interface with and influence memory? Here, inspired by a classic proposal relating perceptual processing to memory durability, the level-of-processing theory, we present a sparse coding model for compressing feature embeddings of images, and show that the reconstruction residuals from this model predict how well images are encoded into memory. In an open memorability dataset of scene images, we show that reconstruction error not only explains memory accuracy, but also response latencies during retrieval, subsuming, in the latter case, all of the variance explained by powerful vision-only models. We also confirm a prediction of this account with ‘model-driven psychophysics’. This work establishes reconstruction error as an important signal interfacing perception and memory, possibly through adaptive modulation of perceptual processing.Keywords
Funding Information
- United States Department of Defense | U.S. Air Force (FA9550-22-1-0041)
This publication has 55 references indexed in Scilit:
- Vision as Bayesian inference: analysis by synthesis?Trends in Cognitive Sciences, 2006
- Levels-of-processing effect on word recognition in schizophreniaBiological Psychiatry, 2003
- Levels of processing: Past, present... and future?Memory, 2002
- Efficient coding of natural soundsNature Neuroscience, 2002
- Observing the transformation of experience into memoryTrends in Cognitive Sciences, 2002
- Building Memories: Remembering and Forgetting of Verbal Experiences as Predicted by Brain ActivityScience, 1998
- Making Memories: Brain Activity that Predicts How Well Visual Experience Will Be RememberedScience, 1998
- Sparse coding with an overcomplete basis set: A strategy employed by V1?Vision Research, 1997
- Emergence of simple-cell receptive field properties by learning a sparse code for natural imagesNature, 1996
- The "Wake-Sleep" Algorithm for Unsupervised Neural NetworksScience, 1995