Rishi Rajalingham
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Rishi Rajalingham has a total of 37 co-authors. Here are the top co-authors by number of shared publications:
- James J. DiCarlo
Massachusetts Institute of Technology
5 shared publications - Kohitij Kar
Department of Biology and Centre for Vision Research Centre for Integrative and Applied Neuroscience York University Toronto Canada
4 shared publications - Mehrdad Jazayeri
Massachusetts Institute of Technology
3 shared publications - Elias B. Issa
Columbia University
2 shared publications - Hansem Sohn
Department of Biomedical Engineering, Sungkyunkwan University (SKKU), Suwon, Republic of Korea
2 shared publications James J DiCarlo
Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139
2 shared publications
Latest Publications
- Open Access
- 2 January 2025
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-16
Dynamic tracking of objects in the macaque dorsomedial frontal cortex
A central tenet of cognitive neuroscience is that humans build an internal model of the external world and use mental simulation of the model to perform physical inferences. Decades of human experiments have shown that behaviors in many physical reasoning tasks are consistent with predictions from the mental simulation theory. However, evidence for the defining feature of mental simulation – that neural population dynamics reflect simulations of physical states in the environment – is limited. We test the mental simulation hypothesis by combining a naturalistic ball-interception task, large-scale electrophysiology in non-human primates, and recurrent neural network modeling. We find that neurons in the monkeys’ dorsomedial frontal cortex (DMFC) represent task-relevant information about the ball position in a multiplexed fashion. At a population level, the activity pattern in DMFC comprises a low-dimensional neural embedding that tracks the ball both when it is visible and invisible, serving as a neural substrate for mental simulation. A systematic comparison of different classes of task-optimized RNN models with the DMFC data provides further evidence supporting the mental simulation hypothesis. Our findings provide evidence that neural dynamics in the frontal cortex are consistent with internal simulation of external states in the environment.
- 2 January 2025
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-16
Open AccessDynamic tracking of objects in the macaque dorsomedial frontal cortex
A central tenet of cognitive neuroscience is that humans build an internal model of the external world and use mental simulation of the model to perform physical inferences. Decades of human experiments have shown that behaviors in many physical reasoning tasks are consistent with predictions from the mental simulation theory. However, evidence for the defining feature of mental simulation – that neural population dynamics reflect simulations of physical states in the environment – is limited. We test the mental simulation hypothesis by combining a naturalistic ball-interception task, large-scale electrophysiology in non-human primates, and recurrent neural network modeling. We find that neurons in the monkeys’ dorsomedial frontal cortex (DMFC) represent task-relevant information about the ball position in a multiplexed fashion. At a population level, the activity pattern in DMFC comprises a low-dimensional neural embedding that tracks the ball both when it is visible and invisible, serving as a neural substrate for mental simulation. A systematic comparison of different classes of task-optimized RNN models with the DMFC data provides further evidence supporting the mental simulation hypothesis. Our findings provide evidence that neural dynamics in the frontal cortex are consistent with internal simulation of external states in the environment.
- Open Access
- 13 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Enabling Advanced Interactions through Closed-loop Control of Motor Unit Activity After Tetraplegia
Studies have shown that most individuals with motor complete spinal cord injuries (SCIs) can activate motor units (MUs) below the injury level, potentially enabling the use of myoelectric controllers for computer tasks. We present a novel wearable neuromotor interface driven by surface electromyography (sEMG). This wearable sEMG interface detects motor unit action potentials (MUAPs) from an array of sensors on the forearm. The MU event rate is translated (i.e. decoded) into discrete or continuous inputs for emulating button presses or joystick inputs. Our study demonstrates real-time detection and decoding of MUAPs in two individuals with tetraplegia, enabling computer task control via a non-invasive interface. Participants calibrated a spike sorting model through periods of rest and tonic activation, mapping MUAP firing rates to various degrees of freedom (DOF) for cursor or character control in tasks and 2D games. Each MUAP was mapped to a separate DOF axis, which we utilized in a Fitts’ Law target acquisition task to evaluate device throughput rate and other performance metrics such as completion rate, initiation time, time to target acquisition, and dial-in time. Our findings highlight MU firing as an effective control input, enabling gaming and social interaction for individuals with tetraplegia. This device’s wearability and ease of use offer an innovative human-computer interaction solution that may enable people with SCI to interact freely with computers and other digital devices.
- 13 October 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessEnabling Advanced Interactions through Closed-loop Control of Motor Unit Activity After Tetraplegia
Studies have shown that most individuals with motor complete spinal cord injuries (SCIs) can activate motor units (MUs) below the injury level, potentially enabling the use of myoelectric controllers for computer tasks. We present a novel wearable neuromotor interface driven by surface electromyography (sEMG). This wearable sEMG interface detects motor unit action potentials (MUAPs) from an array of sensors on the forearm. The MU event rate is translated (i.e. decoded) into discrete or continuous inputs for emulating button presses or joystick inputs. Our study demonstrates real-time detection and decoding of MUAPs in two individuals with tetraplegia, enabling computer task control via a non-invasive interface. Participants calibrated a spike sorting model through periods of rest and tonic activation, mapping MUAP firing rates to various degrees of freedom (DOF) for cursor or character control in tasks and 2D games. Each MUAP was mapped to a separate DOF axis, which we utilized in a Fitts’ Law target acquisition task to evaluate device throughput rate and other performance metrics such as completion rate, initiation time, time to target acquisition, and dial-in time. Our findings highlight MU firing as an effective control input, enabling gaming and social interaction for individuals with tetraplegia. This device’s wearability and ease of use offer an innovative human-computer interaction solution that may enable people with SCI to interact freely with computers and other digital devices.
- Open Access
Dynamic tracking of objects in the macaque dorsomedial frontal cortex
Cognitive neuroscience posits that humans perform physical inferences using mental simulations. Here, we test this hypothesis by analyzing neural activity in the monkeys’ frontal cortex in a ball interception task. We find a low-dimensional neural embedding of the ball position that tracks the ball both when it is visible and invisible. This embedding may serve as a neural substrate for mental simulation.
Open AccessDynamic tracking of objects in the macaque dorsomedial frontal cortex
Cognitive neuroscience posits that humans perform physical inferences using mental simulations. Here, we test this hypothesis by analyzing neural activity in the monkeys’ frontal cortex in a ball interception task. We find a low-dimensional neural embedding of the ball position that tracks the ball both when it is visible and invisible. This embedding may serve as a neural substrate for mental simulation.
- Open Access
The role of mental simulation in primate physical inference abilities
Primates can richly parse sensory inputs to infer latent information, and adjust their behavior accordingly. It has been hypothesized that such flexible inferences are aided by simulations of internal models of the external world. However, evidence supporting this hypothesis has been based on behavioral models that do not emulate neural computations. Here, we test this hypothesis by directly comparing the behavior of humans and monkeys in a ball interception task to that of recurrent neural network (RNN) models with or without the capacity to “simulate” the underlying latent variables. Humans and monkeys had strikingly similar behavioral patterns suggesting common underlying neural computations. Comparison between primates and a large class of RNNs revealed that only RNNs that were optimized to simulate the position of the ball were able to accurately capture key features of the behavior such as systematic biases in the inference process. These results are consistent with the hypothesis that primates use mental simulation to make flexible inferences. Moreover, our work highlights a general strategy for using model neural systems to test computational hypotheses of higher brain function.
Open AccessThe role of mental simulation in primate physical inference abilities
Primates can richly parse sensory inputs to infer latent information, and adjust their behavior accordingly. It has been hypothesized that such flexible inferences are aided by simulations of internal models of the external world. However, evidence supporting this hypothesis has been based on behavioral models that do not emulate neural computations. Here, we test this hypothesis by directly comparing the behavior of humans and monkeys in a ball interception task to that of recurrent neural network (RNN) models with or without the capacity to “simulate” the underlying latent variables. Humans and monkeys had strikingly similar behavioral patterns suggesting common underlying neural computations. Comparison between primates and a large class of RNNs revealed that only RNNs that were optimized to simulate the position of the ball were able to accurately capture key features of the behavior such as systematic biases in the inference process. These results are consistent with the hypothesis that primates use mental simulation to make flexible inferences. Moreover, our work highlights a general strategy for using model neural systems to test computational hypotheses of higher brain function.
- Open Access
Chronically implantable LED arrays for behavioral optogenetics in primates
Challenges in behavioral optogenetics in large brains demand development of a chronically implantable platform for light delivery. We have developed Opto-Array, a chronically implantable array of LEDs for high-throughput optogenetic perturbation in non-human primates. We tested the Opto-Array in the primary visual cortex of a macaque monkey, and demonstrated that optogenetic cortical silencing by the Opto-Array results in reliable retinotopic visual deficits on a luminance discrimination task.
Open AccessChronically implantable LED arrays for behavioral optogenetics in primates
Challenges in behavioral optogenetics in large brains demand development of a chronically implantable platform for light delivery. We have developed Opto-Array, a chronically implantable array of LEDs for high-throughput optogenetic perturbation in non-human primates. We tested the Opto-Array in the primary visual cortex of a macaque monkey, and demonstrated that optogenetic cortical silencing by the Opto-Array results in reliable retinotopic visual deficits on a luminance discrimination task.
- Open Access
- 4 August 2020
- Published by Springer Nature in Nature Communications
- Vol. 11 (1) , 1-13
The inferior temporal cortex is a potential cortical precursor of orthographic processing in untrained monkeys
The ability to recognize written letter strings is foundational to human reading, but the underlying neuronal mechanisms remain largely unknown. Recent behavioral research in baboons suggests that non-human primates may provide an opportunity to investigate this question. We recorded the activity of hundreds of neurons in V4 and the inferior temporal cortex (IT) while naïve macaque monkeys passively viewed images of letters, English words and non-word strings, and tested the capacity of those neuronal representations to support a battery of orthographic processing tasks. We found that simple linear read-outs of IT (but not V4) population responses achieved high performance on all tested tasks, even matching the performance and error patterns of baboons on word classification. These results show that the IT cortex of untrained primates can serve as a precursor of orthographic processing, suggesting that the acquisition of reading in humans relies on the recycling of a brain network evolved for other visual functions.
- 4 August 2020
- Published by Springer Nature in Nature Communications
- Vol. 11 (1) , 1-13
Open AccessThe inferior temporal cortex is a potential cortical precursor of orthographic processing in untrained monkeys
The ability to recognize written letter strings is foundational to human reading, but the underlying neuronal mechanisms remain largely unknown. Recent behavioral research in baboons suggests that non-human primates may provide an opportunity to investigate this question. We recorded the activity of hundreds of neurons in V4 and the inferior temporal cortex (IT) while naïve macaque monkeys passively viewed images of letters, English words and non-word strings, and tested the capacity of those neuronal representations to support a battery of orthographic processing tasks. We found that simple linear read-outs of IT (but not V4) population responses achieved high performance on all tested tasks, even matching the performance and error patterns of baboons on word classification. These results show that the IT cortex of untrained primates can serve as a precursor of orthographic processing, suggesting that the acquisition of reading in humans relies on the recycling of a brain network evolved for other visual functions.
- Open Access
- 6 September 2019
- Published by Association for Research in Vision and Ophthalmology (ARVO) in Journal of Vision
- Vol. 19 (10) , 172b
A precursor of reading: Neural responses to letters strings in the untrained primate inferior temporal cortex predict word recognition behavior
Skilled human readers can readily recognize written letters and words. This domain of visual recognition --- known as orthographic processing --- is critical to human reading abilities, but it is unclear how such behaviors are supported by neural populations in the human brain. A dominant theory is that the existing representations known to support general object recognition, i.e. the ventral visual pathway, could be “recycled” to support orthographic processing. Importantly, recent work has shown that non-human primates (baboons) can learn to distinguish written English words from pseudowords (lexical decision task), successfully generalizing to novel strings, and exhibiting behavioral error patterns that are consistent with humans. These observations demonstrate that non-human primates, while not capturing the entirety of human reading abilities, provide a unique opportunity to investigate the neuronal mechanisms underlying orthographic processing. Here, we investigated the representation of letters and letter strings in the ventral visual stream of naive macaque monkeys, and asked to what extent these representations could support word recognition. We recorded the activity of hundreds of neurons in V4 and IT while monkeys passively viewed images of letters, English words and non-word strings. We then used biologically plausible linear decoders to test these neuronal populations on word recognition. We found that decoders trained on the IT population responses achieved relatively high performance on this lexical decision task. Importantly, the pattern of errors obtained from the IT population was highly correlated with the corresponding primate behavioral pattern. This behavioral pattern was also reproduced by leading artificial neural network models of the ventral stream, but not by low-level representations. Taken together, these results suggest that, even in untrained non-human primates, the population of IT neurons forms an explicit representation of written words that could be “recycled” to support orthographic processing behaviors, serving as a precursor of reading acquisition in humans.
- 6 September 2019
- Published by Association for Research in Vision and Ophthalmology (ARVO) in Journal of Vision
- Vol. 19 (10) , 172b
Open AccessA precursor of reading: Neural responses to letters strings in the untrained primate inferior temporal cortex predict word recognition behavior
Skilled human readers can readily recognize written letters and words. This domain of visual recognition --- known as orthographic processing --- is critical to human reading abilities, but it is unclear how such behaviors are supported by neural populations in the human brain. A dominant theory is that the existing representations known to support general object recognition, i.e. the ventral visual pathway, could be “recycled” to support orthographic processing. Importantly, recent work has shown that non-human primates (baboons) can learn to distinguish written English words from pseudowords (lexical decision task), successfully generalizing to novel strings, and exhibiting behavioral error patterns that are consistent with humans. These observations demonstrate that non-human primates, while not capturing the entirety of human reading abilities, provide a unique opportunity to investigate the neuronal mechanisms underlying orthographic processing. Here, we investigated the representation of letters and letter strings in the ventral visual stream of naive macaque monkeys, and asked to what extent these representations could support word recognition. We recorded the activity of hundreds of neurons in V4 and IT while monkeys passively viewed images of letters, English words and non-word strings. We then used biologically plausible linear decoders to test these neuronal populations on word recognition. We found that decoders trained on the IT population responses achieved relatively high performance on this lexical decision task. Importantly, the pattern of errors obtained from the IT population was highly correlated with the corresponding primate behavioral pattern. This behavioral pattern was also reproduced by leading artificial neural network models of the ventral stream, but not by low-level representations. Taken together, these results suggest that, even in untrained non-human primates, the population of IT neurons forms an explicit representation of written words that could be “recycled” to support orthographic processing behaviors, serving as a precursor of reading acquisition in humans.
- Open Access
A potential cortical precursor of visual word form recognition in untrained monkeys
Skilled human readers can readily recognize written letters and letter strings. This domain of visual recognition, known as orthographic processing, is foundational to human reading, but it is unclear how it is supported by neural populations in the human brain. Behavioral research has shown that non-human primates (baboons) can learn to distinguish written English words from pseudo-words (lexical decision), successfully generalize that behavior to novel strings, and exhibit behavioral error patterns that are consistent with humans. Thus, non-human primate models, while not capturing the entirety of human reading abilities, may provide a unique opportunity to investigate the neuronal mechanisms underlying orthographic processing. Here, we investigated the neuronal representation of letters and letter strings in the ventral visual stream of naive macaque monkeys, and asked to what extent these representations could support visual word recognition. We recorded the activity of hundreds of neurons at the top two levels of the ventral visual form processing pathway (V4 and IT) while monkeys passively viewed images of letters, English words, and non-word letter strings. Linear decoders were used to probe whether those neural responses could support a battery of orthographic processing tasks such as invariant letter identification and lexical decision. We found that IT-based decoders achieved baboon-level performance on these tasks, with a pattern of errors highly correlated to the previously reported primate behavior. This capacity to support orthographic processing tasks was also present in the high-layer units of state-of-the-art artificial neural network models of the ventral stream, but not in the low-layer representations of those models. Taken together, these results show that the IT cortex of untrained monkeys contains a reservoir of precursor features from which downstream brain regions could, with some supervised instruction, learn to support the visual recognition of written words. This suggests that the acquisition of reading in humans did not require a full rebuild of visual processing, but rather the recycling of a brain network evolved for other visual functions.
Open AccessA potential cortical precursor of visual word form recognition in untrained monkeys
Skilled human readers can readily recognize written letters and letter strings. This domain of visual recognition, known as orthographic processing, is foundational to human reading, but it is unclear how it is supported by neural populations in the human brain. Behavioral research has shown that non-human primates (baboons) can learn to distinguish written English words from pseudo-words (lexical decision), successfully generalize that behavior to novel strings, and exhibit behavioral error patterns that are consistent with humans. Thus, non-human primate models, while not capturing the entirety of human reading abilities, may provide a unique opportunity to investigate the neuronal mechanisms underlying orthographic processing. Here, we investigated the neuronal representation of letters and letter strings in the ventral visual stream of naive macaque monkeys, and asked to what extent these representations could support visual word recognition. We recorded the activity of hundreds of neurons at the top two levels of the ventral visual form processing pathway (V4 and IT) while monkeys passively viewed images of letters, English words, and non-word letter strings. Linear decoders were used to probe whether those neural responses could support a battery of orthographic processing tasks such as invariant letter identification and lexical decision. We found that IT-based decoders achieved baboon-level performance on these tasks, with a pattern of errors highly correlated to the previously reported primate behavior. This capacity to support orthographic processing tasks was also present in the high-layer units of state-of-the-art artificial neural network models of the ventral stream, but not in the low-layer representations of those models. Taken together, these results show that the IT cortex of untrained monkeys contains a reservoir of precursor features from which downstream brain regions could, with some supervised instruction, learn to support the visual recognition of written words. This suggests that the acquisition of reading in humans did not require a full rebuild of visual processing, but rather the recycling of a brain network evolved for other visual functions.
- Open Access
Reversible inactivation of different millimeter-scale regions of primate IT results in different patterns of core object recognition deficits
Extensive research suggests that the inferior temporal (IT) population supports visual object recognition behavior. However, causal evidence for this hypothesis has been equivocal, particularly beyond the specific case of face-selective sub-regions of IT. Here, we directly tested this hypothesis by pharmacologically inactivating individual, millimeter-scale sub-regions of IT while monkeys performed several object discrimination tasks, interleaved trial-by-trial. First, we observed that IT inactivation resulted in reliable contralateral-biased task-selective behavioral deficits. Moreover, inactivating different IT sub-regions resulted in different patterns of task deficits, each predicted by that sub-region’s neuronal object discriminability. Finally, the similarity between different inactivation effects was tightly related to the anatomical distance between corresponding inactivation sites. Taken together, these results provide direct evidence that IT cortex causally supports general core object recognition, and that the underlying IT codes are topographically organized.
Open AccessReversible inactivation of different millimeter-scale regions of primate IT results in different patterns of core object recognition deficits
Extensive research suggests that the inferior temporal (IT) population supports visual object recognition behavior. However, causal evidence for this hypothesis has been equivocal, particularly beyond the specific case of face-selective sub-regions of IT. Here, we directly tested this hypothesis by pharmacologically inactivating individual, millimeter-scale sub-regions of IT while monkeys performed several object discrimination tasks, interleaved trial-by-trial. First, we observed that IT inactivation resulted in reliable contralateral-biased task-selective behavioral deficits. Moreover, inactivating different IT sub-regions resulted in different patterns of task deficits, each predicted by that sub-region’s neuronal object discriminability. Finally, the similarity between different inactivation effects was tightly related to the anatomical distance between corresponding inactivation sites. Taken together, these results provide direct evidence that IT cortex causally supports general core object recognition, and that the underlying IT codes are topographically organized.
- Top Cited Papers
- 13 July 2018
- Published by Society for Neuroscience in The Journal of Neuroscience
- Vol. 38 (33) , 7255-7269
Large-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks
Primates, including humans, can typically recognize objects in visual images at a glance despite naturally occurring identity-preserving image transformations (e.g., changes in viewpoint). A primary neuroscience goal is to uncover neuron-level mechanistic models that quantitatively explain this behavior by predicting primate performance for each and every image. Here, we applied this stringent behavioral prediction test to the leading mechanistic models of primate vision (specifically, deep, convolutional, artificial neural networks; ANNs) by directly comparing their behavioral signatures against those of humans and rhesus macaque monkeys. Using high-throughput data collection systems for human and monkey psychophysics, we collected more than one million behavioral trials from 1472 anonymous humans and five male macaque monkeys for 2400 images over 276 binary object discrimination tasks. Consistent with previous work, we observed that state-of-the-art deep, feedforward convolutional ANNs trained for visual categorization (termed DCNNIC models) accurately predicted primate patterns of object-level confusion. However, when we examined behavioral performance for individual images within each object discrimination task, we found that all tested DCNNIC models were significantly nonpredictive of primate performance and that this prediction failure was not accounted for by simple image attributes nor rescued by simple model modifications. These results show that current DCNNIC models cannot account for the image-level behavioral patterns of primates and that new ANN models are needed to more precisely capture the neural mechanisms underlying primate object vision. To this end, large-scale, high-resolution primate behavioral benchmarks such as those obtained here could serve as direct guides for discovering such models. SIGNIFICANCE STATEMENT Recently, specific feedforward deep convolutional artificial neural networks (ANNs) models have dramatically advanced our quantitative understanding of the neural mechanisms underlying primate core object recognition. In this work, we tested the limits of those ANNs by systematically comparing the behavioral responses of these models with the behavioral responses of humans and monkeys at the resolution of individual images. Using these high-resolution metrics, we found that all tested ANN models significantly diverged from primate behavior. Going forward, these high-resolution, large-scale primate behavioral benchmarks could serve as direct guides for discovering better ANN models of the primate visual system.
- 13 July 2018
- Published by Society for Neuroscience in The Journal of Neuroscience
- Vol. 38 (33) , 7255-7269
Top Cited PapersLarge-Scale, High-Resolution Comparison of the Core Visual Object Recognition Behavior of Humans, Monkeys, and State-of-the-Art Deep Artificial Neural Networks
Primates, including humans, can typically recognize objects in visual images at a glance despite naturally occurring identity-preserving image transformations (e.g., changes in viewpoint). A primary neuroscience goal is to uncover neuron-level mechanistic models that quantitatively explain this behavior by predicting primate performance for each and every image. Here, we applied this stringent behavioral prediction test to the leading mechanistic models of primate vision (specifically, deep, convolutional, artificial neural networks; ANNs) by directly comparing their behavioral signatures against those of humans and rhesus macaque monkeys. Using high-throughput data collection systems for human and monkey psychophysics, we collected more than one million behavioral trials from 1472 anonymous humans and five male macaque monkeys for 2400 images over 276 binary object discrimination tasks. Consistent with previous work, we observed that state-of-the-art deep, feedforward convolutional ANNs trained for visual categorization (termed DCNNIC models) accurately predicted primate patterns of object-level confusion. However, when we examined behavioral performance for individual images within each object discrimination task, we found that all tested DCNNIC models were significantly nonpredictive of primate performance and that this prediction failure was not accounted for by simple image attributes nor rescued by simple model modifications. These results show that current DCNNIC models cannot account for the image-level behavioral patterns of primates and that new ANN models are needed to more precisely capture the neural mechanisms underlying primate object vision. To this end, large-scale, high-resolution primate behavioral benchmarks such as those obtained here could serve as direct guides for discovering such models. SIGNIFICANCE STATEMENT Recently, specific feedforward deep convolutional artificial neural networks (ANNs) models have dramatically advanced our quantitative understanding of the neural mechanisms underlying primate core object recognition. In this work, we tested the limits of those ANNs by systematically comparing the behavioral responses of these models with the behavioral responses of humans and monkeys at the resolution of individual images. Using these high-resolution metrics, we found that all tested ANN models significantly diverged from primate behavior. Going forward, these high-resolution, large-scale primate behavioral benchmarks could serve as direct guides for discovering better ANN models of the primate visual system.