Abstract
Inter-individual variability in fine-grained functional topographies poses challenges for scalable data analysis and modeling. Functional alignment techniques can help mitigate these individual differences but they typically require paired brain data with the same stimuli between individuals, which are often unavailable. Here we present a neural code conversion method that overcomes this constraint by optimizing conversion parameters based on the discrepancy between the stimulus contents represented by original and converted brain activity patterns. This approach, combined with hierarchical features of deep neural networks as latent content representations, achieves conversion accuracies that are comparable with methods using shared stimuli. The converted brain activity from a source subject can be accurately decoded using the target’s pre-trained decoders, producing high-quality visual image reconstructions that rival within-individual decoding, even with data across different sites and limited training samples. Our approach offers a promising framework for scalable neural data analysis and modeling and a foundation for brain-to-brain communication.
Funding Information
  • MEXT | Japan Society for the Promotion of Science (KAKENHI grants JP20H05705 and JP20H05954 to Y.K., KAKENHI grants JP20H05705 and JP20H05954 to Y.K.)
  • Japan Agency for Medical Research and Development (grant JP24wm0625409 to Y.K., grant JP24wm0625409 to Y.K.)
  • MEXT | Japan Science and Technology Agency (CREST grants JPMJCR18A5 and JPMJCR22P3 to Y.K., CREST grants JPMJCR18A5 and JPMJCR22P3 to Y.K.)
  • New Energy and Industrial Technology Development Organization (grant JPNP20006 to Y.K., grant JPNP20006 to Y.K.)
  • Guardian Robot Project, RIKEN