Images of the unseen: extrapolating visual representations for abstract and concrete words in a data-driven computational model
Open Access
- 12 November 2020
- journal article
- research article
- Published by Springer Nature in Psychological Research
- Vol. 86 (8) , 2512-2532
Abstract
Theories of grounded cognition assume that conceptual representations are grounded in sensorimotor experience. However, abstract concepts such as jealousy or childhood have no directly associated referents with which such sensorimotor experience can be made; therefore, the grounding of abstract concepts has long been a topic of debate. Here, we propose (a) that systematic relations exist between semantic representations learned from language on the one hand and perceptual experience on the other hand, (b) that these relations can be learned in a bottom-up fashion, and (c) that it is possible to extrapolate from this learning experience to predict expected perceptual representations for words even where direct experience is missing. To test this, we implement a data-driven computational model that is trained to map language-based representations (obtained from text corpora, representing language experience) onto vision-based representations (obtained from an image database, representing perceptual experience), and apply its mapping function onto language-based representations for abstract and concrete words outside the training set. In three experiments, we present participants with these words, accompanied by two images: the image predicted by the model and a random control image. Results show that participants’ judgements were in line with model predictions even for the most abstract words. This preference was stronger for more concrete items and decreased for the more abstract ones. Taken together, our findings have substantial implications in support of the grounding of abstract words, suggesting that we can tap into our previous experience to create possible visual representation we don’t have.Keywords
Funding Information
- Deutsche Forschungsgemeinschaft (392225719)
- Fondazione Cariplo (2017-1633)
- Eberhard Karls Universität Tübingen
This publication has 93 references indexed in Scilit:
- Random effects structure for confirmatory hypothesis testing: Keep it maximalJournal of Memory and Language, 2013
- A hierarchical view of grounded, embodied, and situated numerical cognitionCognitive Processing, 2012
- Neuroplasticity of semantic representations for musical instruments in professional musiciansNeuroImage, 2011
- On the need for Embodied and Dis-Embodied CognitionFrontiers in Psychology, 2011
- Language Encodes Geographical InformationCognitive Science, 2009
- Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed modelsJournal of Memory and Language, 2008
- High-dimensional semantic space accounts of primingJournal of Memory and Language, 2006
- Perceptions of perceptual symbolsBehavioral and Brain Sciences, 1999
- Minds, brains, and programsBehavioral and Brain Sciences, 1980
- The Metaphorical Structure of the Human Conceptual SystemCognitive Science, 1980