Accurately decoding visual information from fMRI data obtained in a realistic virtual environment
Authors:
Open Access
- 9 June 2015
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Human Neuroscience
- Vol. 9, 327
Abstract
Three-dimensional interactive virtual environments are a powerful tool for brain-imaging based cognitive neuroscience that are presently under-utilized. This paper presents machine-learning based methods for identifying brain states induced by realistic virtual environments with improved accuracy as well as the capability for mapping their spatial topography on the neocortex. Virtual environments provide the ability to study the brain under conditions closer to the environment in which humans evolved, and thus to probe deeper into the complexities of human cognition. As a test case, we designed a stimulus to reflect a military combat situation in the Middle East, motivated by the potential of using real-time functional magnetic resonance imaging (fMRI) in the treatment of post-traumatic stress disorder. Each subject experienced moving through the virtual town where they encountered 1—6 animated combatants at different locations, while fMRI data was collected. To analyze the data from what is, compared to most studies, more complex and less controlled stimuli, we employed statistical machine learning in the form of Multi-Voxel Pattern Analysis (MVPA) with special attention given to artificial Neural Networks (NN). Extensions to NN that exploit the block structure of the stimulus were developed to improve the accuracy of the classification, achieving performances from 58%—93% (chance was 16.7%) with 6 subjects. This demonstrates that MVPA can decode a complex cognitive state, viewing a number of characters, in a dynamic virtual environment. To better understand the source of this information in the brain, a novel form of sensitivity analysis was developed to use NN to quantify the degree to which each voxel contributed to classification. Compared with maps produced by general linear models and the searchlight approach, these sensitivity maps revealed a more diverse pattern of information relevant to the classification of cognitive state.Keywords
This publication has 48 references indexed in Scilit:
- Efficacy of Virtual Reality Exposure Therapy in the Treatment of PTSD: A Systematic ReviewPLOS ONE, 2012
- A virtual reality-based FMRI study of reward-based spatial learningNeuropsychologia, 2010
- Machine learning classifiers and fMRI: A tutorial overviewNeuroImage, 2009
- PyMVPA: a Python Toolbox for Multivariate Pattern Analysis of fMRI DataNeuroinformatics, 2009
- "Who" Is Saying "What"? Brain-Based Decoding of Human Voice and SpeechScience, 2008
- Relating Retinotopic and Object-Selective Responses in Human Lateral Occipital CortexJournal of Neurophysiology, 2008
- Using fMRI Brain Activation to Identify Cognitive States Associated with Perception of Tools and DwellingsPLOS ONE, 2008
- Neural substrates of driving behaviourNeuroImage, 2007
- Beyond mind-reading: multi-voxel pattern analysis of fMRI dataTrends in Cognitive Sciences, 2006
- Distributed and Overlapping Representations of Faces and Objects in Ventral Temporal CortexScience, 2001