Kshitij Dwivedi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Kshitij Dwivedi has a total of 193 co-authors. Here are the top co-authors by number of shared publications:
- Gemma Roig
Goethe University Frankfurt
8 shared publications - Radoslaw Martin Cichy
Neural Dynamics of Visual Cognition Group, Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
5 shared publications - Alessandro Thomas Gifford
Institute of Psychology, Freie Universität Berlin, Berlin, Germany
2 shared publications - Alexandre Hyafil
Centre de Recerca Matemàtica
2 shared publications - Arvind Kumar
Department of Computational Science and Technology, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 11428 Stockholm, Sweden
2 shared publications - Athena Akrami
Sainsbury Wellcome Centre, UCL
2 shared publications
Latest Publications
- Open Access
- 24 July 2024
- Published by Springer Nature in Nature Communications
- Vol. 15 (1) , 1-26
Modeling short visual events through the BOLD moments video fMRI dataset and metadata
Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos’ extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.
- 24 July 2024
- Published by Springer Nature in Nature Communications
- Vol. 15 (1) , 1-26
Open AccessModeling short visual events through the BOLD moments video fMRI dataset and metadata
Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos’ extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.
- Open Access
What do navigation agents learn about their environment?
Today's state of the art visual navigation agents typically consist of large deep learning models trained end to end. Such models offer little to no interpretability about the learned skills or the actions of the agent taken in response to its environment. While past works have explored interpreting deep learning models, little attention has been devoted to interpreting embodied AI systems, which often involve reasoning about the structure of the environment, target characteristics and the outcome of one's actions. In this paper, we introduce the Interpretability System for Embodied agEnts (iSEE) for Point Goal and Object Goal navigation agents. We use iSEE to probe the dynamic representations produced by these agents for the presence of information about the agent as well as the environment. We demonstrate interesting insights about navigation agents using iSEE, including the ability to encode reachable locations (to avoid obstacles), visibility of the target, progress from the initial spawn location as well as the dramatic effect on the behaviors of agents when we mask out critical individual neurons. The code is available at: https://github.com/allenai/iSEE
Open AccessWhat do navigation agents learn about their environment?
Today's state of the art visual navigation agents typically consist of large deep learning models trained end to end. Such models offer little to no interpretability about the learned skills or the actions of the agent taken in response to its environment. While past works have explored interpreting deep learning models, little attention has been devoted to interpreting embodied AI systems, which often involve reasoning about the structure of the environment, target characteristics and the outcome of one's actions. In this paper, we introduce the Interpretability System for Embodied agEnts (iSEE) for Point Goal and Object Goal navigation agents. We use iSEE to probe the dynamic representations produced by these agents for the presence of information about the agent as well as the environment. We demonstrate interesting insights about navigation agents using iSEE, including the ability to encode reachable locations (to avoid obstacles), visibility of the target, progress from the initial spawn location as well as the dramatic effect on the behaviors of agents when we mask out critical individual neurons. The code is available at: https://github.com/allenai/iSEE
- Open Access
- 31 March 2022
- Published by The Open Journal in Journal of Open Source Education
- Vol. 5 (49) , 118
Neuromatch Academy: a 3-week, online summer school in computational neuroscience
- 31 March 2022
- Published by The Open Journal in Journal of Open Source Education
- Vol. 5 (49) , 118
Open AccessNeuromatch Academy: a 3-week, online summer school in computational neuroscience
- Open Access
A large and rich EEG dataset for modeling human visual object recognition
The human brain achieves visual object recognition through multiple stages of nonlinear transformations operating at a millisecond scale. To predict and explain these rapid transformations, computational neuroscientists employ machine learning modeling techniques. However, state-of-the-art models require massive amounts of data to properly train, and to the present day there is a lack of vast brain datasets which extensively sample the temporal dynamics of visual object recognition. Here we collected a large and rich dataset of high temporal resolution EEG responses to images of objects on a natural background. This dataset includes 10 participants, each with 82,160 trials spanning 16,740 image conditions. Through computational modeling we established the quality of this dataset in five ways. First, we trained linearizing encoding models that successfully synthesized the EEG responses to arbitrary images. Second, we correctly identified the recorded EEG data image conditions in a zero-shot fashion, using EEG synthesized responses to hundreds of thousands of candidate image conditions. Third, we show that both the high number of conditions as well as the trial repetitions of the EEG dataset contribute to the trained models’ prediction accuracy. Fourth, we built encoding models whose predictions well generalize to novel participants. Fifth, we demonstrate full end-to-end training of randomly initialized DNNs that output M/EEG responses for arbitrary input images. We release this dataset as a tool to foster research in visual neuroscience and computer vision.
Open AccessA large and rich EEG dataset for modeling human visual object recognition
The human brain achieves visual object recognition through multiple stages of nonlinear transformations operating at a millisecond scale. To predict and explain these rapid transformations, computational neuroscientists employ machine learning modeling techniques. However, state-of-the-art models require massive amounts of data to properly train, and to the present day there is a lack of vast brain datasets which extensively sample the temporal dynamics of visual object recognition. Here we collected a large and rich dataset of high temporal resolution EEG responses to images of objects on a natural background. This dataset includes 10 participants, each with 82,160 trials spanning 16,740 image conditions. Through computational modeling we established the quality of this dataset in five ways. First, we trained linearizing encoding models that successfully synthesized the EEG responses to arbitrary images. Second, we correctly identified the recorded EEG data image conditions in a zero-shot fashion, using EEG synthesized responses to hundreds of thousands of candidate image conditions. Third, we show that both the high number of conditions as well as the trial repetitions of the EEG dataset contribute to the trained models’ prediction accuracy. Fourth, we built encoding models whose predictions well generalize to novel participants. Fifth, we demonstrate full end-to-end training of randomly initialized DNNs that output M/EEG responses for arbitrary input images. We release this dataset as a tool to foster research in visual neuroscience and computer vision.
- Open Access
- 24 February 2022
- Published by Springer Nature in Nature Human Behaviour
- Vol. 6 (6) , 796-811
The spatiotemporal neural dynamics of object location representations in the human brain
To interact with objects in complex environments, we must know what they are and where they are in spite of challenging viewing conditions. Here, we investigated where, how and when representations of object location and category emerge in the human brain when objects appear on cluttered natural scene images using a combination of functional magnetic resonance imaging, electroencephalography and computational models. We found location representations to emerge along the ventral visual stream towards lateral occipital complex, mirrored by gradual emergence in deep neural networks. Time-resolved analysis suggested that computing object location representations involves recurrent processing in high-level visual cortex. Object category representations also emerged gradually along the ventral visual stream, with evidence for recurrent computations. These results resolve the spatiotemporal dynamics of the ventral visual stream that give rise to representations of where and what objects are present in a scene under challenging viewing conditions.
- 24 February 2022
- Published by Springer Nature in Nature Human Behaviour
- Vol. 6 (6) , 796-811
Open AccessThe spatiotemporal neural dynamics of object location representations in the human brain
To interact with objects in complex environments, we must know what they are and where they are in spite of challenging viewing conditions. Here, we investigated where, how and when representations of object location and category emerge in the human brain when objects appear on cluttered natural scene images using a combination of functional magnetic resonance imaging, electroencephalography and computational models. We found location representations to emerge along the ventral visual stream towards lateral occipital complex, mirrored by gradual emergence in deep neural networks. Time-resolved analysis suggested that computing object location representations involves recurrent processing in high-level visual cortex. Object category representations also emerged gradually along the ventral visual stream, with evidence for recurrent computations. These results resolve the spatiotemporal dynamics of the ventral visual stream that give rise to representations of where and what objects are present in a scene under challenging viewing conditions.
- Open Access
- 13 August 2021
- Published by Public Library of Science (PLoS) in PLOS Computational Biology
- Vol. 17 (8) , e1009267
Unveiling functions of the visual cortex using task-specific deep neural networks
The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.
- 13 August 2021
- Published by Public Library of Science (PLoS) in PLOS Computational Biology
- Vol. 17 (8) , e1009267
Open AccessUnveiling functions of the visual cortex using task-specific deep neural networks
The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.
- Open Access
- 15 February 2021
- Published by Center for Open Science in OSF Preprints
Neuromatch Academy: a 3-week, online summer school in computational neuroscience
Neuromatch Academy (https://neuromatch.io/academy) was designed as an online summer school to cover the basics of computational neuroscience in three weeks. The materials cover dominant and emerging computational neuroscience tools, how they complement one another, and specifically focus on how they can help us to better understand how the brain functions. An original component of the materials is its focus on modeling choices, i.e. how do we choose the right approach, how do we build models, and how can we evaluate models to determine if they provide real (meaningful) insight. This meta-modeling component of the instructional materials asks what questions can be answered by different techniques, and how to apply them meaningfully to get insight about brain function.
- 15 February 2021
- Published by Center for Open Science in OSF Preprints
Open AccessNeuromatch Academy: a 3-week, online summer school in computational neuroscience
Neuromatch Academy (https://neuromatch.io/academy) was designed as an online summer school to cover the basics of computational neuroscience in three weeks. The materials cover dominant and emerging computational neuroscience tools, how they complement one another, and specifically focus on how they can help us to better understand how the brain functions. An original component of the materials is its focus on modeling choices, i.e. how do we choose the right approach, how do we build models, and how can we evaluate models to determine if they provide real (meaningful) insight. This meta-modeling component of the instructional materials asks what questions can be answered by different techniques, and how to apply them meaningfully to get insight about brain function.
- Open Access
Unveiling functions of the visual cortex using task-specific deep neural networks
The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.
Open AccessUnveiling functions of the visual cortex using task-specific deep neural networks
The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.
- 13 November 2020
- Published by Springer Nature
- p. 497-513
Duality Diagram Similarity: A Generic Framework for Initialization Selection in Task Transfer Learning
- 13 November 2020
- Published by Springer Nature
- p. 497-513
Duality Diagram Similarity: A Generic Framework for Initialization Selection in Task Transfer Learning
- Open Access
- 6 September 2019
- Published by Association for Research in Vision and Ophthalmology (ARVO) in Journal of Vision
- Vol. 19 (10) , 190b
Explaining Scene-selective Visual Area Using Task-specific and Category-specific DNN Units
Deep neural networks (DNN) trained for classification are often used to explain responses of the visual cortex. Recently it was demonstrated that a DNN trained on a task related to the function of a brain region explains its responses better than a DNN trained on a task which is not explicitly related. Taking motivation from the previous results, in this work we investigate if we can infer the functionality of different areas in the scene-selective visual cortex by comparing the correlation of brain areas with DNNs trained on different tasks. We select 20 DNNs trained on diverse computer vision tasks including multiple 2D, 3D, and semantic tasks from the Taskonomy dataset. We select 2 areas in the scene-selective visual cortex, namely occipital place area (OPA) and parahippocampal place area (PPA). We perform representation similarity analysis (RSA) of OPA and PPA with 20 DNNs to investigate if the relative correlation of brain areas with different tasks can shed some light into the functions of these brain areas. The results reveal that OPA shows a higher correlation with 3D tasks as compared to semantic and 2D tasks, while PPA shows a higher correlation with semantic tasks. We further probe the functionality of these brain areas by category-specific units of a scene-parsing DNN. The results reveal that OPA shows high correlation with the categorical units crucial for navigational affordances while PPA shows high correlation with the categorical units crucial for scene-classification. Our results are consistent with previous neuroimaging studies investigating the function of PPA and OPA showing that PPA is involved in scene classification while OPA is involved in representing 3D scene structure and navigational affordances. Our results suggest that performing a searchlight analysis with RDMs of DNNs trained on different tasks may reveal the functional map of the visual cortex.
- 6 September 2019
- Published by Association for Research in Vision and Ophthalmology (ARVO) in Journal of Vision
- Vol. 19 (10) , 190b
Open AccessExplaining Scene-selective Visual Area Using Task-specific and Category-specific DNN Units
Deep neural networks (DNN) trained for classification are often used to explain responses of the visual cortex. Recently it was demonstrated that a DNN trained on a task related to the function of a brain region explains its responses better than a DNN trained on a task which is not explicitly related. Taking motivation from the previous results, in this work we investigate if we can infer the functionality of different areas in the scene-selective visual cortex by comparing the correlation of brain areas with DNNs trained on different tasks. We select 20 DNNs trained on diverse computer vision tasks including multiple 2D, 3D, and semantic tasks from the Taskonomy dataset. We select 2 areas in the scene-selective visual cortex, namely occipital place area (OPA) and parahippocampal place area (PPA). We perform representation similarity analysis (RSA) of OPA and PPA with 20 DNNs to investigate if the relative correlation of brain areas with different tasks can shed some light into the functions of these brain areas. The results reveal that OPA shows a higher correlation with 3D tasks as compared to semantic and 2D tasks, while PPA shows a higher correlation with semantic tasks. We further probe the functionality of these brain areas by category-specific units of a scene-parsing DNN. The results reveal that OPA shows high correlation with the categorical units crucial for navigational affordances while PPA shows high correlation with the categorical units crucial for scene-classification. Our results are consistent with previous neuroimaging studies investigating the function of PPA and OPA showing that PPA is involved in scene classification while OPA is involved in representing 3D scene structure and navigational affordances. Our results suggest that performing a searchlight analysis with RDMs of DNNs trained on different tasks may reveal the functional map of the visual cortex.