Decoding Realistic Images from Brain Activity with Contrastive Self-Supervision and Latent Diffusion
Authors:
Open Access
- 28 September 2023
- book chapter
- Published by IOS Press
- p. 2250-2257
Abstract
Reconstructing visual stimuli from human brain activities provides a promising opportunity to advance our understanding of the brain’s visual system and its connection with computer vision models. Although deep generative models have been employed for this task, the challenge of generating high-quality images with accurate semantics persists due to the intricate underlying representations of brain signals and the limited availability of parallel data. In this paper, we propose a two-phase framework named Contrast and Diffuse (CnD) to decode realistic images from functional magnetic resonance imaging (fMRI) recordings. In the first phase, we acquire representations of fMRI data through self-supervised contrastive learning. In the second phase, the encoded fMRI representations condition the diffusion model to reconstruct visual stimulus through our proposed concept-aware conditioning method. Experimental results show that CnD reconstructs highly plausible images on challenging benchmarks. We also provide a quantitative interpretation of the connection between the latent diffusion model (LDM) components and the human brain’s visual system. In summary, we present an effective approach for reconstructing visual stimuli based on human brain activity and offer a novel framework to understand the relationship between the diffusion model and the human brain visual system. The code is released at https://github.com/Mingxiao-Li/BrainDecoding.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- High-resolution image reconstruction with latent diffusion models from human brain activity2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- An fMRI Dataset for Concept Representation with Semantic Feature AnnotationsScientific Data, 2022
- Self-supervised Natural Image Reconstruction and Large-scale Semantic Classification from Brain ActivityNeuroImage, 2022
- Neural Encoding and Decoding With Distributed Sentence RepresentationsIEEE Transactions on Neural Networks and Learning Systems, 2021
- The Neural Representation of Visually Evoked Emotion Is High-Dimensional, Categorical, and Distributed across Transmodal Brain RegionsiScience, 2020
- Generative adversarial networks for reconstructing natural images from brain activityNeuroImage, 2018
- Generic decoding of seen and imagined objects using hierarchical visual featuresNature Communications, 2017
- Identifying natural images from human brain activityNature, 2008