Generating Text from Functional Brain Images
Open Access
- 1 January 2011
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Human Neuroscience
- Vol. 5, 11244
Abstract
Recent work has shown that it is possible to take brain images acquired during viewing of a scene and reconstruct an approximation of the scene from those images. Here we show that it is also possible to generate text about the mental content reflected in brain images. We began with images collected as participants read names of concrete items (e.g., "Apartment") while also seeing line drawings of the item named. We built a model of the mental semantic representation of concrete concepts from text data and learned to map aspects of such representation to patterns of activation in the corresponding brain image. In order to validate this mapping, without accessing information about the items viewed for left-out individual brain images, we were able to generate from each one a collection of semantically pertinent words (e.g., "door," "window" for "Apartment"). Furthermore, we show that the ability to generate such words allows us to perform a classification task and thus validate our method quantitatively.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Visual Image Reconstruction from Human Brain Activity using a Combination of Multiscale Local Image DecodersNeuron, 2008
- Predicting Human Brain Activity Associated with the Meanings of NounsScience, 2008
- Identifying natural images from human brain activityNature, 2008
- Inverse retinotopy: Inferring the visual content of images from brain activation patternsNeuroImage, 2006
- Breaking the silence: Brain–computer interfaces (BCI) for communication and motor controlPsychophysiology, 2006
- Beyond mind-reading: multi-voxel pattern analysis of fMRI dataTrends in Cognitive Sciences, 2006
- Decoding mental states from brain activity in humansNature Reviews Neuroscience, 2006
- Classifying spatial patterns of brain activity with machine learning methods: Application to lie detectionNeuroImage, 2005
- A standardized set of 260 pictures: Norms for name agreement, image agreement, familiarity, and visual complexity.Journal of Experimental Psychology: Human Learning and Memory, 1980
- Concreteness, imagery, and meaningfulness values for 925 nouns.Journal of Experimental Psychology, 1968