Sound reconstruction from human brain activity via a generative model with brain-like auditory features
Authors:
- Jong-Yun ParkJPJong-Yun Park
- Mitsuaki TsukamotoMTMitsuaki Tsukamoto
- Misato TanakaMTMisato Tanaka
- Yukiyasu KamitaniYKYukiyasu Kamitani
Preprint
Abstract
The successful reconstruction of perceptual experiences from human brain activity has provided insights into the neural representations of sensory experiences. However, reconstructing arbitrary sounds has been avoided due to the complexity of temporal sequences in sounds and the limited resolution of neuroimaging modalities. To overcome these challenges, leveraging the hierarchical nature of brain auditory processing could provide a path toward reconstructing arbitrary sounds. Previous studies have indicated a hierarchical homology between the human auditory system and deep neural network (DNN) models. Furthermore, advancements in audio-generative models enable to transform compressed representations back into high-resolution sounds. In this study, we introduce a novel sound reconstruction method that combines brain decoding of auditory features with an audio-generative model. Using fMRI responses to natural sounds, we found that the hierarchical sound features of a DNN model could be better decoded than spectrotemporal features. We then reconstructed the sound using an audio transformer that disentangled compressed temporal information in the decoded DNN features. Our method shows unconstrained sounds reconstruction capturing sound perceptual contents and quality and generalizability by reconstructing sound categories not included in the training dataset. Reconstructions from different auditory regions remain similar to actual sounds, highlighting the distributed nature of auditory representations. To see whether the reconstructions mirrored actual subjective perceptual experiences, we performed an experiment involving selective auditory attention to one of overlapping sounds. The results tended to resemble the attended sound than the unattended. These findings demonstrate that our proposed model provides a means to externalize experienced auditory contents from human brain activity.Keywords
All Related Versions
This publication has 48 references indexed in Scilit:
- Decoding neurobiological spike trains using recurrent neural networks: a case study with electrophysiological auditory cortex recordingsNeural Computing and Applications, 2021
- Parallel and distributed encoding of speech across human auditory cortexCell, 2021
- The Neural Representation of Visually Evoked Emotion Is High-Dimensional, Categorical, and Distributed across Transmodal Brain RegionsiScience, 2020
- Where is the cocktail party? Decoding locations of attended and unattended moving sound sources using EEGNeuroImage, 2020
- Interpretable, highly accurate brain decoding of subtly distinct brain states from functional MRI using intrinsic functional networks and long short-term memory recurrent neural networksNeuroImage, 2019
- Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortexNeuroImage, 2019
- Decoding fMRI activity in the time domain improves classification performanceNeuroImage, 2018
- Progress in speech decoding from the electrocorticogramBiomedical Engineering Letters, 2015
- Functional organization of human auditory cortex: Investigation of response latencies through direct recordingsNeuroImage, 2014
- The cocktail party problemCurrent Biology, 2009