Raphael T. Gerraty
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Raphael T. Gerraty has a total of 57 co-authors. Here are the top co-authors by number of shared publications:
- Daphna Shohamy
Department of Psychology, Columbia University, New York, NY, USA
5 shared publications - Danielle S. Bassett
Yale University
4 shared publications Kosha Ruparel
Neuropsychiatry Section
4 shared publications- Camilla van Geen
Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104
3 shared publications James Loughead
Brain Behavior Laboratory Neuropsychiatry Department Hospital of the University of Pennsylvania Philadelphia PA USA
3 shared publicationsMark A. Elliott
Department of Radiology, University of Pennsylvania Perelman School of Medicine
3 shared publications
Latest Publications
- Open Access
Two routes to value-based decisions in Parkinson’s disease: differentiating incremental reinforcement learning from episodic memory
Patients with Parkinson’s disease are impaired at incremental reward-based learning. It is typically assumed that this impairment reflects a loss of striatal dopamine. However, many open questions remain about the nature of reward-based learning deficits in Parkinson’s. Recent studies have found that a combination of different cognitive and computational strategies contribute even to simple reward-based learning tasks, suggesting a possible role for episodic memory. These findings raise critical questions about how incremental learning and episodic memory interact to support learning from past experience and what their relative contributions are to impaired decision-making in Parkinson’s disease. Here we addressed these questions by asking patients with Parkinson’s disease (n=26) both on and off their dopamine replacement medication and age- and education-matched healthy controls (n=26) to complete a task designed to isolate the contributions of incremental learning and episodic memory to reward-based learning and decision-making. We found that Parkinson’s patients performed as well as healthy controls when using episodic memory, but were impaired at incremental reward-based learning. Dopamine replacement medication remediated this deficit while enhancing subsequent episodic memory for the value of motivationally relevant stimuli. These results demonstrate that Parkinson’s patients are impaired at learning about reward from trial-and-error when episodic memory is properly controlled for, and that learning based on the value of single experiences remains intact in patients with Parkinson’s disease.
Open AccessTwo routes to value-based decisions in Parkinson’s disease: differentiating incremental reinforcement learning from episodic memory
Patients with Parkinson’s disease are impaired at incremental reward-based learning. It is typically assumed that this impairment reflects a loss of striatal dopamine. However, many open questions remain about the nature of reward-based learning deficits in Parkinson’s. Recent studies have found that a combination of different cognitive and computational strategies contribute even to simple reward-based learning tasks, suggesting a possible role for episodic memory. These findings raise critical questions about how incremental learning and episodic memory interact to support learning from past experience and what their relative contributions are to impaired decision-making in Parkinson’s disease. Here we addressed these questions by asking patients with Parkinson’s disease (n=26) both on and off their dopamine replacement medication and age- and education-matched healthy controls (n=26) to complete a task designed to isolate the contributions of incremental learning and episodic memory to reward-based learning and decision-making. We found that Parkinson’s patients performed as well as healthy controls when using episodic memory, but were impaired at incremental reward-based learning. Dopamine replacement medication remediated this deficit while enhancing subsequent episodic memory for the value of motivationally relevant stimuli. These results demonstrate that Parkinson’s patients are impaired at learning about reward from trial-and-error when episodic memory is properly controlled for, and that learning based on the value of single experiences remains intact in patients with Parkinson’s disease.
- Open Access
Source Invariance and Probabilistic Transfer: A Testable Theory of Probabilistic Neural Representations
As animals interact with their environments, they must infer properties of their surroundings. Some animals, including humans, can represent uncertainty about those properties. But when, if ever, do they use probability distributions to represent their uncertainty? It depends on which definition we choose. In this paper, we argue that existing definitions are inadequate because they are untestable. We then propose our own definition. There are two reasons why existing definitions are untestable. First, they do not distinguish between representations of uncertainty and representations of variables merely related to uncertainty ('representational indeterminacy'). Second, they do not distinguish between probabilistic representations of uncertainty and merely "heuristic" representations of uncertainty. We call this 'model indeterminacy' because the underlying problem is that we do not have access to the animal's generative model. We define probabilistic representations by two properties: 1) they encode uncertainty regardless of the source of the uncertainty ('source invariance'), 2) they support the efficient learning of new tasks that would be more difficult to learn given non-probabilistic representations ('probabilistic task transfer'). Source invariance indicates that they are representations of uncertainty rather than variables merely related to uncertainty, thereby solving representational indeterminacy. Probabilistic task transfer indicates that they are probabilistic representations of uncertainty rather than merely heuristic representations, thereby solving model indeterminacy.
Open AccessSource Invariance and Probabilistic Transfer: A Testable Theory of Probabilistic Neural Representations
As animals interact with their environments, they must infer properties of their surroundings. Some animals, including humans, can represent uncertainty about those properties. But when, if ever, do they use probability distributions to represent their uncertainty? It depends on which definition we choose. In this paper, we argue that existing definitions are inadequate because they are untestable. We then propose our own definition. There are two reasons why existing definitions are untestable. First, they do not distinguish between representations of uncertainty and representations of variables merely related to uncertainty ('representational indeterminacy'). Second, they do not distinguish between probabilistic representations of uncertainty and merely "heuristic" representations of uncertainty. We call this 'model indeterminacy' because the underlying problem is that we do not have access to the animal's generative model. We define probabilistic representations by two properties: 1) they encode uncertainty regardless of the source of the uncertainty ('source invariance'), 2) they support the efficient learning of new tasks that would be more difficult to learn given non-probabilistic representations ('probabilistic task transfer'). Source invariance indicates that they are representations of uncertainty rather than variables merely related to uncertainty, thereby solving representational indeterminacy. Probabilistic task transfer indicates that they are probabilistic representations of uncertainty rather than merely heuristic representations, thereby solving model indeterminacy.
- Open Access
- 21 July 2022
- Published by Wiley in Human Brain Mapping
- Vol. 43 (15) , 4750-4790
Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T
The model‐free algorithms of “reinforcement learning” (RL) have gained clout across disciplines, but so too have model‐based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This “generalized reinforcement learning” (GRL) model, a frugal extension of RL, parsimoniously retains the single reward‐prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal‐learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high‐resolution high‐field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value‐based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.
- 21 July 2022
- Published by Wiley in Human Brain Mapping
- Vol. 43 (15) , 4750-4790
Open AccessReinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T
The model‐free algorithms of “reinforcement learning” (RL) have gained clout across disciplines, but so too have model‐based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This “generalized reinforcement learning” (GRL) model, a frugal extension of RL, parsimoniously retains the single reward‐prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal‐learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high‐resolution high‐field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value‐based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.
- 1 December 2021
- Published by Elsevier in Journal of Mathematical Psychology
- Vol. 105
Hierarchical Bayesian models of reinforcement learning: Introduction and comparison to alternative methods
- 1 December 2021
- Published by Elsevier in Journal of Mathematical Psychology
- Vol. 105
Hierarchical Bayesian models of reinforcement learning: Introduction and comparison to alternative methods
- Open Access
- 11 October 2021
- Published by Center for Open Science in PsyArXiv
Probabilistic representations should be defined by their computational role: reply to Rahnev et al. 2021
Note: This is a reply to the Generative Adversarial Collaboration "Is perception probabilistic? Clarifying the definitions" by Rahnev, Block, Denison, and Jehee (DOI 10.31234/osf.io/f8v5r). Probabilistic representations promise to shed light on how humans and other organisms cope with uncertainty, but debates about them have been hindered by unclear definitions and a lack of testable predictions. We argue that neither camp in this GAC has offered a plausible way out of this situation. We describe problems with their attempts to define probabilistic representations and sketch an alternative view, in which the defining feature of a probabilistic representation is how it is used.
- 11 October 2021
- Published by Center for Open Science in PsyArXiv
Open AccessProbabilistic representations should be defined by their computational role: reply to Rahnev et al. 2021
Note: This is a reply to the Generative Adversarial Collaboration "Is perception probabilistic? Clarifying the definitions" by Rahnev, Block, Denison, and Jehee (DOI 10.31234/osf.io/f8v5r). Probabilistic representations promise to shed light on how humans and other organisms cope with uncertainty, but debates about them have been hindered by unclear definitions and a lack of testable predictions. We argue that neither camp in this GAC has offered a plausible way out of this situation. We describe problems with their attempts to define probabilistic representations and sketch an alternative view, in which the defining feature of a probabilistic representation is how it is used.
- Open Access
- 20 October 2020
- Published by Cold Spring Harbor Laboratory in bioRxiv
Hierarchical Bayesian Models of Reinforcement Learning: Introduction and comparison to alternative methods
Reinforcement learning models have been used extensively to capture learning and decision-making processes in humans and other organisms. One essential goal of these computational models is the generalization to new sets of observations. Extracting parameters that can reliably predict out-of-sample data can be difficult, however. The use of prior distributions to regularize parameter estimates has been shown to help remedy this issue. While previous research has suggested that empirical priors estimated from a separate dataset improve predictive accuracy, this paper outlines an alternate method for the derivation of empirical priors: hierarchical Bayesian modeling. We provide a detailed introduction to this method, and show that using hierarchical models to simultaneously extract and impose empirical priors leads to better out-of-sample prediction while being more data efficient.
- 20 October 2020
- Published by Cold Spring Harbor Laboratory in bioRxiv
Open AccessHierarchical Bayesian Models of Reinforcement Learning: Introduction and comparison to alternative methods
Reinforcement learning models have been used extensively to capture learning and decision-making processes in humans and other organisms. One essential goal of these computational models is the generalization to new sets of observations. Extracting parameters that can reliably predict out-of-sample data can be difficult, however. The use of prior distributions to regularize parameter estimates has been shown to help remedy this issue. While previous research has suggested that empirical priors estimated from a separate dataset improve predictive accuracy, this paper outlines an alternate method for the derivation of empirical priors: hierarchical Bayesian modeling. We provide a detailed introduction to this method, and show that using hierarchical models to simultaneously extract and impose empirical priors leads to better out-of-sample prediction while being more data efficient.
- Open Access
Dopamine modulates learning-related changes in dynamic striatal-cortical connectivity in Parkinson’s disease
Learning from reinforcement is thought to depend on striatal dopamine inputs, which serve to update the value of actions by modifying connections in widespread cortico-striatal circuits. While considerable research has described the activity of individual striatal and midbrain regions in reinforcement learning, the broader role for dopamine in modulating network-level processes has been difficult to decipher. To examine whether dopamine modulates circuit-level dynamic connectivity during learning, we characterized the effects of dopamine on learning-related dynamic functional connectivity estimated from fMRI data acquired in patients with Parkinson’s disease. Patients with Parkinson’s disease have severe dopamine depletion in the striatum and are treated with dopamine replacement drugs, providing an opportunity to compare learning and network dynamics when patients are in a low dopamine state (off drugs) versus a high dopamine state (on drugs). We assessed the relationship between dopamine and dynamic connectivity while patients performed a probabilistic reversal learning task. We found that reversal learning altered dynamic network flexibility in the striatum and that this effect was dependent on dopaminergic state. We also found that dopamine modulated changes in connectivity between the striatum and specific task-relevant visual areas of inferior temporal cortex, providing empirical support for theories stipulating that value is updated through changes in cortico-striatal circuits. These results suggest that dopamine exerts a widespread effect on neural circuitry and network dynamics during reinforcement learning.
Open AccessDopamine modulates learning-related changes in dynamic striatal-cortical connectivity in Parkinson’s disease
Learning from reinforcement is thought to depend on striatal dopamine inputs, which serve to update the value of actions by modifying connections in widespread cortico-striatal circuits. While considerable research has described the activity of individual striatal and midbrain regions in reinforcement learning, the broader role for dopamine in modulating network-level processes has been difficult to decipher. To examine whether dopamine modulates circuit-level dynamic connectivity during learning, we characterized the effects of dopamine on learning-related dynamic functional connectivity estimated from fMRI data acquired in patients with Parkinson’s disease. Patients with Parkinson’s disease have severe dopamine depletion in the striatum and are treated with dopamine replacement drugs, providing an opportunity to compare learning and network dynamics when patients are in a low dopamine state (off drugs) versus a high dopamine state (on drugs). We assessed the relationship between dopamine and dynamic connectivity while patients performed a probabilistic reversal learning task. We found that reversal learning altered dynamic network flexibility in the striatum and that this effect was dependent on dopaminergic state. We also found that dopamine modulated changes in connectivity between the striatum and specific task-relevant visual areas of inferior temporal cortex, providing empirical support for theories stipulating that value is updated through changes in cortico-striatal circuits. These results suggest that dopamine exerts a widespread effect on neural circuitry and network dynamics during reinforcement learning.
- Open Access
- 5 February 2018
- Published by Society for Neuroscience in The Journal of Neuroscience
- Vol. 38 (10) , 2442-2453
Dynamic Flexibility in Striatal-Cortical Circuits Supports Reinforcement Learning
Complex learned behaviors must involve the integrated action of distributed brain circuits. Although the contributions of individual regions to learning have been extensively investigated, much less is known about how distributed brain networks orchestrate their activity over the course of learning. To address this gap, we used fMRI combined with tools from dynamic network neuroscience to obtain time-resolved descriptions of network coordination during reinforcement learning in humans. We found that learning to associate visual cues with reward involves dynamic changes in network coupling between the striatum and distributed brain regions, including visual, orbitofrontal, and ventromedial prefrontal cortex (n= 22; 13 females). Moreover, we found that this flexibility in striatal network coupling correlates with participants' learning rate and inverse temperature, two parameters derived from reinforcement learning models. Finally, we found that episodic learning, measured separately in the same participants at the same time, was related to dynamic connectivity in distinct brain networks. These results suggest that dynamic changes in striatal-centered networks provide a mechanism for information integration during reinforcement learning.SIGNIFICANCE STATEMENTLearning from the outcomes of actions, referred to asreinforcement learning, is an essential part of life. The roles of individual brain regions in reinforcement learning have been well characterized in terms of updating values for actions or cues. Missing from this account, however, is an understanding of how different brain areas interact during learning to integrate sensory and value information. Here we characterize flexible striatal-cortical network dynamics that relate to reinforcement learning behavior.
- 5 February 2018
- Published by Society for Neuroscience in The Journal of Neuroscience
- Vol. 38 (10) , 2442-2453
Open AccessDynamic Flexibility in Striatal-Cortical Circuits Supports Reinforcement Learning
Complex learned behaviors must involve the integrated action of distributed brain circuits. Although the contributions of individual regions to learning have been extensively investigated, much less is known about how distributed brain networks orchestrate their activity over the course of learning. To address this gap, we used fMRI combined with tools from dynamic network neuroscience to obtain time-resolved descriptions of network coordination during reinforcement learning in humans. We found that learning to associate visual cues with reward involves dynamic changes in network coupling between the striatum and distributed brain regions, including visual, orbitofrontal, and ventromedial prefrontal cortex (n= 22; 13 females). Moreover, we found that this flexibility in striatal network coupling correlates with participants' learning rate and inverse temperature, two parameters derived from reinforcement learning models. Finally, we found that episodic learning, measured separately in the same participants at the same time, was related to dynamic connectivity in distinct brain networks. These results suggest that dynamic changes in striatal-centered networks provide a mechanism for information integration during reinforcement learning.SIGNIFICANCE STATEMENTLearning from the outcomes of actions, referred to asreinforcement learning, is an essential part of life. The roles of individual brain regions in reinforcement learning have been well characterized in terms of updating values for actions or cues. Missing from this account, however, is an understanding of how different brain areas interact during learning to integrate sensory and value information. Here we characterize flexible striatal-cortical network dynamics that relate to reinforcement learning behavior.
- Open Access
Dynamic flexibility in striatal-cortical circuits supports reinforcement learning
Complex learned behaviors must involve the integrated action of distributed brain circuits. While the contributions of individual regions to learning have been extensively investigated, understanding how distributed brain networks orchestrate their activity over the course of learning remains elusive. To address this gap, we used fMRI combined with tools from dynamic network neuroscience to obtain time-resolved descriptions of network coordination during reinforcement learning. We found that learning to associate visual cues with reward involves dynamic changes in network coupling between the striatum and distributed brain regions, including visual, orbitofrontal, and ventromedial prefrontal cortex. Moreover, we found that flexibility in striatal network dynamics correlates with participants’ learning rate and inverse temperature, two parameters derived from reinforcement learning models. Finally, we found that not all forms of learning relate to this circuit: episodic memory, measured in the same participants at the same time, was related to dynamic connectivity in distinct brain networks. These results suggest that dynamic changes in striatal-centered networks provide a mechanism for information integration during reinforcement learning. Significance Statement: Learning from the outcomes of actions–referred to as reinforcement learning –is an essential part of life. The roles of individual brain regions in reinforcement learning have been well characterized in terms of the updating of values for actions or sensory stimuli. Missing from this account, however, is a description of the manner in which different brain areas interact during learning to integrate sensory and value information. Here we characterize flexible striatal-cortical network dynamics that relate to reinforcement learning behavior.
Open AccessDynamic flexibility in striatal-cortical circuits supports reinforcement learning
Complex learned behaviors must involve the integrated action of distributed brain circuits. While the contributions of individual regions to learning have been extensively investigated, understanding how distributed brain networks orchestrate their activity over the course of learning remains elusive. To address this gap, we used fMRI combined with tools from dynamic network neuroscience to obtain time-resolved descriptions of network coordination during reinforcement learning. We found that learning to associate visual cues with reward involves dynamic changes in network coupling between the striatum and distributed brain regions, including visual, orbitofrontal, and ventromedial prefrontal cortex. Moreover, we found that flexibility in striatal network dynamics correlates with participants’ learning rate and inverse temperature, two parameters derived from reinforcement learning models. Finally, we found that not all forms of learning relate to this circuit: episodic memory, measured in the same participants at the same time, was related to dynamic connectivity in distinct brain networks. These results suggest that dynamic changes in striatal-centered networks provide a mechanism for information integration during reinforcement learning. Significance Statement: Learning from the outcomes of actions–referred to as reinforcement learning –is an essential part of life. The roles of individual brain regions in reinforcement learning have been well characterized in terms of the updating of values for actions or sensory stimuli. Missing from this account, however, is a description of the manner in which different brain areas interact during learning to integrate sensory and value information. Here we characterize flexible striatal-cortical network dynamics that relate to reinforcement learning behavior.
- Open Access
- 20 August 2014
- Published by Society for Neuroscience in Journal of Neuroscience
- Vol. 34 (34) , 11297-11303
Transfer of Learning Relates to Intrinsic Connectivity between Hippocampus, Ventromedial Prefrontal Cortex, and Large-Scale Networks
An important aspect of adaptive learning is the ability to flexibly use past experiences to guide new decisions. When facing a new decision, some people automatically leverage previously learned associations, while others do not. This variability in transfer of learning across individuals has been demonstrated repeatedly and has important implications for understanding adaptive behavior, yet the source of these individual differences remains poorly understood. In particular, it is unknown why such variability in transfer emerges even among homogeneous groups of young healthy participants who do not vary on other learning-related measures. Here we hypothesized that individual differences in the transfer of learning could be related to relatively stable differences in intrinsic brain connectivity, which could constrain how individuals learn. To test this, we obtained a behavioral measure of memory-based transfer outside of the scanner and on a separate day acquired resting-state functional MRI images in 42 participants. We then analyzed connectivity across independent component analysis-derived brain networks during rest, and tested whether intrinsic connectivity in learning-related networks was associated with transfer. We found that individual differences in transfer were related to intrinsic connectivity between the hippocampus and the ventromedial prefrontal cortex, and between these regions and large-scale functional brain networks. Together, the findings demonstrate a novel role for intrinsic brain dynamics in flexible learning-guided behavior, both within a set of functionally specific regions known to be important for learning, as well as between these regions and the default and frontoparietal networks, which are thought to serve more general cognitive functions.
- 20 August 2014
- Published by Society for Neuroscience in Journal of Neuroscience
- Vol. 34 (34) , 11297-11303
Open AccessTransfer of Learning Relates to Intrinsic Connectivity between Hippocampus, Ventromedial Prefrontal Cortex, and Large-Scale Networks
An important aspect of adaptive learning is the ability to flexibly use past experiences to guide new decisions. When facing a new decision, some people automatically leverage previously learned associations, while others do not. This variability in transfer of learning across individuals has been demonstrated repeatedly and has important implications for understanding adaptive behavior, yet the source of these individual differences remains poorly understood. In particular, it is unknown why such variability in transfer emerges even among homogeneous groups of young healthy participants who do not vary on other learning-related measures. Here we hypothesized that individual differences in the transfer of learning could be related to relatively stable differences in intrinsic brain connectivity, which could constrain how individuals learn. To test this, we obtained a behavioral measure of memory-based transfer outside of the scanner and on a separate day acquired resting-state functional MRI images in 42 participants. We then analyzed connectivity across independent component analysis-derived brain networks during rest, and tested whether intrinsic connectivity in learning-related networks was associated with transfer. We found that individual differences in transfer were related to intrinsic connectivity between the hippocampus and the ventromedial prefrontal cortex, and between these regions and large-scale functional brain networks. Together, the findings demonstrate a novel role for intrinsic brain dynamics in flexible learning-guided behavior, both within a set of functionally specific regions known to be important for learning, as well as between these regions and the default and frontoparietal networks, which are thought to serve more general cognitive functions.