KDKshitij Dwivedi

Kshitij Dwivedi

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Kshitij Dwivedi has a total of 193 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Modeling short visual events through the BOLD moments video fMRI dataset and metadata

    by Benjamin Lahner, Kshitij Dwivedi, Polina Iamshchinina, Monika Graumann, Alex Lascelles, Gemma Roig, Alessandro Thomas Gifford, Bowen Pan, SouYoung Jin, N. Apurva Ratan Murty, et al.

    Studying the neural basis of human dynamic visual perception requires extensive experimental data to evaluate the large swathes of functionally diverse brain neural networks driven by perceiving visual events. Here, we introduce the BOLD Moments Dataset (BMD), a repository of whole-brain fMRI responses to over 1000 short (3 s) naturalistic video clips of visual events across ten human subjects. We use the videos’ extensive metadata to show how the brain represents word- and sentence-level descriptions of visual events and identify correlates of video memorability scores extending into the parietal cortex. Furthermore, we reveal a match in hierarchical processing between cortical regions of interest and video-computable deep neural networks, and we showcase that BMD successfully captures temporal dynamics of visual events at second resolution. With its rich metadata, BMD offers new perspectives and accelerates research on the human brain basis of visual event perception.


  • Open Access

    What do navigation agents learn about their environment?

    by Kshitij Dwivedi, Gemma Roig, Aniruddha Kembhavi, Roozbeh Mottaghi

    Today's state of the art visual navigation agents typically consist of large deep learning models trained end to end. Such models offer little to no interpretability about the learned skills or the actions of the agent taken in response to its environment. While past works have explored interpreting deep learning models, little attention has been devoted to interpreting embodied AI systems, which often involve reasoning about the structure of the environment, target characteristics and the outcome of one's actions. In this paper, we introduce the Interpretability System for Embodied agEnts (iSEE) for Point Goal and Object Goal navigation agents. We use iSEE to probe the dynamic representations produced by these agents for the presence of information about the agent as well as the environment. We demonstrate interesting insights about navigation agents using iSEE, including the ability to encode reachable locations (to avoid obstacles), visibility of the target, progress from the initial spawn location as well as the dramatic effect on the behaviors of agents when we mask out critical individual neurons. The code is available at: https://github.com/allenai/iSEE

  • Open Access

    Neuromatch Academy: a 3-week, online summer school in computational neuroscience

    by Bernard Marius 'T Hart, Titipat Achakulvisut, Ayoade Adeyemi, Athena Akrami, Bradly Alicea, Alicia Alonso-Andres, Diego Alzate-Correa, Arash Ash, Jesus J. Ballesteros, Aishwarya Balwani, et al.


  • Open Access

    A large and rich EEG dataset for modeling human visual object recognition

    by Alessandro T. Gifford, Kshitij Dwivedi, Gemma Roig, Radoslaw M. Cichy

    The human brain achieves visual object recognition through multiple stages of nonlinear transformations operating at a millisecond scale. To predict and explain these rapid transformations, computational neuroscientists employ machine learning modeling techniques. However, state-of-the-art models require massive amounts of data to properly train, and to the present day there is a lack of vast brain datasets which extensively sample the temporal dynamics of visual object recognition. Here we collected a large and rich dataset of high temporal resolution EEG responses to images of objects on a natural background. This dataset includes 10 participants, each with 82,160 trials spanning 16,740 image conditions. Through computational modeling we established the quality of this dataset in five ways. First, we trained linearizing encoding models that successfully synthesized the EEG responses to arbitrary images. Second, we correctly identified the recorded EEG data image conditions in a zero-shot fashion, using EEG synthesized responses to hundreds of thousands of candidate image conditions. Third, we show that both the high number of conditions as well as the trial repetitions of the EEG dataset contribute to the trained models’ prediction accuracy. Fourth, we built encoding models whose predictions well generalize to novel participants. Fifth, we demonstrate full end-to-end training of randomly initialized DNNs that output M/EEG responses for arbitrary input images. We release this dataset as a tool to foster research in visual neuroscience and computer vision.


  • Open Access

    The spatiotemporal neural dynamics of object location representations in the human brain

    by Monika Graumann, Caterina Ciuffi, Kshitij Dwivedi, Gemma Roig, Radoslaw M. Cichy

    To interact with objects in complex environments, we must know what they are and where they are in spite of challenging viewing conditions. Here, we investigated where, how and when representations of object location and category emerge in the human brain when objects appear on cluttered natural scene images using a combination of functional magnetic resonance imaging, electroencephalography and computational models. We found location representations to emerge along the ventral visual stream towards lateral occipital complex, mirrored by gradual emergence in deep neural networks. Time-resolved analysis suggested that computing object location representations involves recurrent processing in high-level visual cortex. Object category representations also emerged gradually along the ventral visual stream, with evidence for recurrent computations. These results resolve the spatiotemporal dynamics of the ventral visual stream that give rise to representations of where and what objects are present in a scene under challenging viewing conditions.


  • Open Access

    Unveiling functions of the visual cortex using task-specific deep neural networks

    by Kshitij Dwivedi, Michael F. Bonner, Radoslaw Martin Cichy, Gemma Roig

    The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.


  • Open Access

    Neuromatch Academy: a 3-week, online summer school in computational neuroscience

    by Bernard Marius 'T Hart, Titipat Achakulvisut, Athena Akrami, Bradly Alicea, Ulrik Beierholm, Gunnar Blohm, Kathryn Bonnen, John S Butler, Brandon Caie, You Cheng, et al.

    Neuromatch Academy (https://neuromatch.io/academy) was designed as an online summer school to cover the basics of computational neuroscience in three weeks. The materials cover dominant and emerging computational neuroscience tools, how they complement one another, and specifically focus on how they can help us to better understand how the brain functions. An original component of the materials is its focus on modeling choices, i.e. how do we choose the right approach, how do we build models, and how can we evaluate models to determine if they provide real (meaningful) insight. This meta-modeling component of the instructional materials asks what questions can be answered by different techniques, and how to apply them meaningfully to get insight about brain function.


  • Open Access

    Unveiling functions of the visual cortex using task-specific deep neural networks

    by Kshitij Dwivedi, Michael F. Bonner, Radoslaw Martin Cichy, Gemma Roig

    The human visual cortex enables visual perception through a cascade of hierarchical computations in cortical regions with distinct functionalities. Here, we introduce an AI-driven approach to discover the functional mapping of the visual cortex. We related human brain responses to scene images measured with functional MRI (fMRI) systematically to a diverse set of deep neural networks (DNNs) optimized to perform different scene perception tasks. We found a structured mapping between DNN tasks and brain regions along the ventral and dorsal visual streams. Low-level visual tasks mapped onto early brain regions, 3-dimensional scene perception tasks mapped onto the dorsal stream, and semantic tasks mapped onto the ventral stream. This mapping was of high fidelity, with more than 60% of the explainable variance in nine key regions being explained. Together, our results provide a novel functional mapping of the human visual cortex and demonstrate the power of the computational approach.

  • Open Access

    Explaining Scene-selective Visual Area Using Task-specific and Category-specific DNN Units

    by Kshitij Dwivedi, Michael F Bonner, Gemma Roig

    Deep neural networks (DNN) trained for classification are often used to explain responses of the visual cortex. Recently it was demonstrated that a DNN trained on a task related to the function of a brain region explains its responses better than a DNN trained on a task which is not explicitly related. Taking motivation from the previous results, in this work we investigate if we can infer the functionality of different areas in the scene-selective visual cortex by comparing the correlation of brain areas with DNNs trained on different tasks. We select 20 DNNs trained on diverse computer vision tasks including multiple 2D, 3D, and semantic tasks from the Taskonomy dataset. We select 2 areas in the scene-selective visual cortex, namely occipital place area (OPA) and parahippocampal place area (PPA). We perform representation similarity analysis (RSA) of OPA and PPA with 20 DNNs to investigate if the relative correlation of brain areas with different tasks can shed some light into the functions of these brain areas. The results reveal that OPA shows a higher correlation with 3D tasks as compared to semantic and 2D tasks, while PPA shows a higher correlation with semantic tasks. We further probe the functionality of these brain areas by category-specific units of a scene-parsing DNN. The results reveal that OPA shows high correlation with the categorical units crucial for navigational affordances while PPA shows high correlation with the categorical units crucial for scene-classification. Our results are consistent with previous neuroimaging studies investigating the function of PPA and OPA showing that PPA is involved in scene classification while OPA is involved in representing 3D scene structure and navigational affordances. Our results suggest that performing a searchlight analysis with RDMs of DNNs trained on different tasks may reveal the functional map of the visual cortex.