Samuel J. Gershman
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Samuel J. Gershman has a total of 20 co-authors. Here are the top co-authors by number of shared publications:
- Kenneth Andrew Norman
Princeton Neuroscience Institute, Princeton University, Princeton, USA
3 shared publications - David M. Blei
Department of Computer Science, Princeton University, 35 Olden Street, Princeton, NJ 08540, USA
2 shared publications Yael Niv
Princeton Neuroscience Institute and Department of Psychology, Princeton University
2 shared publications- Ahmed Abdelhalim Moustafa
Bond University
1 shared publication Almut Hupbach
Department of Psychology, Lehigh University, Bethlehem, Pennsylvania, 18015
1 shared publicationAngela Radulescu
Department of Psychology and Princeton Neuroscience Institute, Princeton University, Princeton, New Jersey, United States of America
1 shared publication
Latest Publications
- 1 September 2016
- Published by Elsevier in Vision Research
- Vol. 126, 232-241
Discovering hierarchical motion structure
- 1 September 2016
- Published by Elsevier in Vision Research
- Vol. 126, 232-241
Discovering hierarchical motion structure
- 23 June 2015
- Published by Springer Nature in Learning & Behavior
- Vol. 43 (3) , 243-250
Individual differences in learning predict the return of fear
Using a laboratory analogue of learned fear (Pavlovian fear conditioning), we show that there is substantial heterogeneity across individuals in spontaneous recovery of fear following extinction training. We propose that this heterogeneity might stem from qualitative individual differences in the nature of extinction learning. Whereas some individuals tend to form a new memory during extinction, leaving their fear memory intact, others update the original threat association with new safety information, effectively unlearning the fear memory. We formalize this account in a computational model of fear learning and show that individuals who, according to the model, are more likely to form new extinction memories tend to show greater spontaneous recovery compared to individuals who appear to only update a single memory. This qualitative variation in fear and extinction learning may have important implications for understanding vulnerability and resilience to fear-related psychiatric disorders.
- 23 June 2015
- Published by Springer Nature in Learning & Behavior
- Vol. 43 (3) , 243-250
Individual differences in learning predict the return of fear
Using a laboratory analogue of learned fear (Pavlovian fear conditioning), we show that there is substantial heterogeneity across individuals in spontaneous recovery of fear following extinction training. We propose that this heterogeneity might stem from qualitative individual differences in the nature of extinction learning. Whereas some individuals tend to form a new memory during extinction, leaving their fear memory intact, others update the original threat association with new safety information, effectively unlearning the fear memory. We formalize this account in a computational model of fear learning and show that individuals who, according to the model, are more likely to form new extinction memories tend to show greater spontaneous recovery compared to individuals who appear to only update a single memory. This qualitative variation in fear and extinction learning may have important implications for understanding vulnerability and resilience to fear-related psychiatric disorders.
- Open Access
- 23 March 2015
- Published by Wiley in Topics in Cognitive Science
- Vol. 7 (3) , 391-415
Novelty and Inductive Generalization in Human Reinforcement Learning
In reinforcement learning (RL), a decision maker searching for the most rewarding option is often faced with the question: What is the value of an option that has never been tried before? One way to frame this question is as an inductive problem: How can I generalize my previous experience with one set of options to a novel option? We show how hierarchical Bayesian inference can be used to solve this problem, and we describe an equivalence between the Bayesian model and temporal difference learning algorithms that have been proposed as models of RL in humans and animals. According to our view, the search for the best option is guided by abstract knowledge about the relationships between different options in an environment, resulting in greater search efficiency compared to traditional RL algorithms previously applied to human cognition. In two behavioral experiments, we test several predictions of our model, providing evidence that humans learn and exploit structured inductive knowledge to make predictions about novel options. In light of this model, we suggest a new interpretation of dopaminergic responses to novelty.
- 23 March 2015
- Published by Wiley in Topics in Cognitive Science
- Vol. 7 (3) , 391-415
Open AccessNovelty and Inductive Generalization in Human Reinforcement Learning
In reinforcement learning (RL), a decision maker searching for the most rewarding option is often faced with the question: What is the value of an option that has never been tried before? One way to frame this question is as an inductive problem: How can I generalize my previous experience with one set of options to a novel option? We show how hierarchical Bayesian inference can be used to solve this problem, and we describe an equivalence between the Bayesian model and temporal difference learning algorithms that have been proposed as models of RL in humans and animals. According to our view, the search for the best option is guided by abstract knowledge about the relationships between different options in an environment, resulting in greater search efficiency compared to traditional RL algorithms previously applied to human cognition. In two behavioral experiments, we test several predictions of our model, providing evidence that humans learn and exploit structured inductive knowledge to make predictions about novel options. In light of this model, we suggest a new interpretation of dopaminergic responses to novelty.
- 13 January 2015
- Published by Springer Nature in Psychonomic Bulletin & Review
- Vol. 22 (5) , 1320-1327
Do learning rates adapt to the distribution of rewards?
Studies of reinforcement learning have shown that humans learn differently in response to positive and negative reward prediction errors, a phenomenon that can be captured computationally by positing asymmetric learning rates. This asymmetry, motivated by neurobiological and cognitive considerations, has been invoked to explain learning differences across the lifespan as well as a range of psychiatric disorders. Recent theoretical work, motivated by normative considerations, has hypothesized that the learning rate asymmetry should be modulated by the distribution of rewards across the available options. In particular, the learning rate for negative prediction errors should be higher than the learning rate for positive prediction errors when the average reward rate is high, and this relationship should reverse when the reward rate is low. We tested this hypothesis in a series of experiments. Contrary to the theoretical predictions, we found that the asymmetry was largely insensitive to the average reward rate; instead, the dominant pattern was a higher learning rate for negative than for positive prediction errors, possibly reflecting risk aversion.
- 13 January 2015
- Published by Springer Nature in Psychonomic Bulletin & Review
- Vol. 22 (5) , 1320-1327
Do learning rates adapt to the distribution of rewards?
Studies of reinforcement learning have shown that humans learn differently in response to positive and negative reward prediction errors, a phenomenon that can be captured computationally by positing asymmetric learning rates. This asymmetry, motivated by neurobiological and cognitive considerations, has been invoked to explain learning differences across the lifespan as well as a range of psychiatric disorders. Recent theoretical work, motivated by normative considerations, has hypothesized that the learning rate asymmetry should be modulated by the distribution of rewards across the available options. In particular, the learning rate for negative prediction errors should be higher than the learning rate for positive prediction errors when the average reward rate is high, and this relationship should reverse when the reward rate is low. We tested this hypothesis in a series of experiments. Contrary to the theoretical predictions, we found that the asymmetry was largely insensitive to the average reward rate; instead, the dominant pattern was a higher learning rate for negative than for positive prediction errors, possibly reflecting risk aversion.
- Open Access
- 6 November 2014
- Published by Public Library of Science (PLoS) in PLOS Computational Biology
- Vol. 10 (11) , e1003939
Statistical Computations Underlying the Dynamics of Memory Updating
Psychophysical and neurophysiological studies have suggested that memory is not simply a carbon copy of our experience: Memories are modified or new memories are formed depending on the dynamic structure of our experience, and specifically, on how gradually or abruptly the world changes. We present a statistical theory of memory formation in a dynamic environment, based on a nonparametric generalization of the switching Kalman filter. We show that this theory can qualitatively account for several psychophysical and neural phenomena, and present results of a new visual memory experiment aimed at testing the theory directly. Our experimental findings suggest that humans can use temporal discontinuities in the structure of the environment to determine when to form new memory traces. The statistical perspective we offer provides a coherent account of the conditions under which new experience is integrated into an old memory versus forming a new memory, and shows that memory formation depends on inferences about the underlying structure of our experience. When do we modify old memories, and when do we create new ones? We suggest that this question can be answered statistically: The parsing of experience into distinct memory traces corresponds to inferences about the underlying structure of the environment. When sensory data change gradually over time, the brain infers that the environment has slowly been evolving, and the current representation of the environment (an existing memory trace) is updated. In contrast, abrupt changes indicate transitions between different structures, leading to the formation of new memories. While these ideas fall naturally out of statistical models of learning, they have not yet been directly tested in the domain of human memory. In this paper, we describe a model of statistical inference that instantiates these ideas, and test the model by asking human participants to reconstruct previously seen visual objects that have since changed gradually or abruptly. The results of this experiment support our theory of how the statistical structure of sensory experiences shapes memory formation.
- 6 November 2014
- Published by Public Library of Science (PLoS) in PLOS Computational Biology
- Vol. 10 (11) , e1003939
Open AccessStatistical Computations Underlying the Dynamics of Memory Updating
Psychophysical and neurophysiological studies have suggested that memory is not simply a carbon copy of our experience: Memories are modified or new memories are formed depending on the dynamic structure of our experience, and specifically, on how gradually or abruptly the world changes. We present a statistical theory of memory formation in a dynamic environment, based on a nonparametric generalization of the switching Kalman filter. We show that this theory can qualitatively account for several psychophysical and neural phenomena, and present results of a new visual memory experiment aimed at testing the theory directly. Our experimental findings suggest that humans can use temporal discontinuities in the structure of the environment to determine when to form new memory traces. The statistical perspective we offer provides a coherent account of the conditions under which new experience is integrated into an old memory versus forming a new memory, and shows that memory formation depends on inferences about the underlying structure of our experience. When do we modify old memories, and when do we create new ones? We suggest that this question can be answered statistically: The parsing of experience into distinct memory traces corresponds to inferences about the underlying structure of the environment. When sensory data change gradually over time, the brain infers that the environment has slowly been evolving, and the current representation of the environment (an existing memory trace) is updated. In contrast, abrupt changes indicate transitions between different structures, leading to the formation of new memories. While these ideas fall naturally out of statistical models of learning, they have not yet been directly tested in the domain of human memory. In this paper, we describe a model of statistical inference that instantiates these ideas, and test the model by asking human participants to reconstruct previously seen visual objects that have since changed gradually or abruptly. The results of this experiment support our theory of how the statistical structure of sensory experiences shapes memory formation.
- 1 September 2014
- Published by Elsevier in NeuroImage
- Vol. 98, 91-102
Decomposing spatiotemporal brain patterns into topographic latent sources
- 1 September 2014
- Published by Elsevier in NeuroImage
- Vol. 98, 91-102
Decomposing spatiotemporal brain patterns into topographic latent sources
- 1 May 2014
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 37 (2) , 334-345
Distance Dependent Infinite Latent Feature Models
Latent feature models are widely used to decompose data into a small number of components. Bayesian nonparametric variants of these models, which use the Indian buffet process (IBP) as a prior over latent features, allow the number of features to be determined from the data. We present a generalization of the IBP, the distance dependent Indian buffet process (dd-IBP), for modeling non-exchangeable data. It relies on distances defined between data points, biasing nearby data to share more features. The choice of distance measure allows for many kinds of dependencies, including temporal and spatial. Further, the original IBP is a special case of the dd-IBP. We develop the dd-IBP and theoretically characterize its feature-sharing properties. We derive a Markov chain Monte Carlo sampler for a linear Gaussian model with a dd-IBP prior and study its performance on real-world non-exchangeable data.
- 1 May 2014
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 37 (2) , 334-345
Distance Dependent Infinite Latent Feature Models
Latent feature models are widely used to decompose data into a small number of components. Bayesian nonparametric variants of these models, which use the Indian buffet process (IBP) as a prior over latent features, allow the number of features to be determined from the data. We present a generalization of the IBP, the distance dependent Indian buffet process (dd-IBP), for modeling non-exchangeable data. It relies on distances defined between data points, biasing nearby data to share more features. The choice of distance measure allows for many kinds of dependencies, including temporal and spatial. Further, the original IBP is a special case of the dd-IBP. We develop the dd-IBP and theoretically characterize its feature-sharing properties. We derive a Markov chain Monte Carlo sampler for a linear Gaussian model with a dd-IBP prior and study its performance on real-world non-exchangeable data.
- 28 March 2014
- Published by Taylor & Francis in Network: Computation in Neural Systems
- Vol. 25 (3) , 97-115
The penumbra of learning: A statistical theory of synaptic tagging and capture
Learning in humans and animals is accompanied by a penumbra: Learning one task benefits from learning an unrelated task shortly before or after. At the cellular level, the penumbra of learning appears when weak potentiation of one synapse is amplified by strong potentiation of another synapse on the same neuron during a critical time window. Weak potentiation sets a molecular tag that enables the synapse to capture plasticity-related proteins synthesized in response to strong potentiation at another synapse. This paper describes a computational model which formalizes synaptic tagging and capture in terms of statistical learning mech- anisms. According to this model, synaptic strength encodes a probabilistic inference about the dynamically changing association between pre- and post-synaptic firing rates. The rate of change is itself inferred, coupling together different synapses on the same neuron. When the inputs to one synapse change rapidly, the inferred rate of change increases, amplifying learning at other synapses.
- 28 March 2014
- Published by Taylor & Francis in Network: Computation in Neural Systems
- Vol. 25 (3) , 97-115
The penumbra of learning: A statistical theory of synaptic tagging and capture
Learning in humans and animals is accompanied by a penumbra: Learning one task benefits from learning an unrelated task shortly before or after. At the cellular level, the penumbra of learning appears when weak potentiation of one synapse is amplified by strong potentiation of another synapse on the same neuron during a critical time window. Weak potentiation sets a molecular tag that enables the synapse to capture plasticity-related proteins synthesized in response to strong potentiation at another synapse. This paper describes a computational model which formalizes synaptic tagging and capture in terms of statistical learning mech- anisms. According to this model, synaptic strength encodes a probabilistic inference about the dynamically changing association between pre- and post-synaptic firing rates. The rate of change is itself inferred, coupling together different synapses on the same neuron. When the inputs to one synapse change rapidly, the inferred rate of change increases, amplifying learning at other synapses.
- 1 March 2014
- Published by MIT Press in Neural Computation
- Vol. 26 (3) , 467-471
Dopamine Ramps Are a Consequence of Reward Prediction Errors
Temporal difference learning models of dopamine assert that phasic levels of dopamine encode a reward prediction error. However, this hypothesis has been challenged by recent observations of gradually ramping stratal dopamine levels as a goal is approached. This note describes conditions under which temporal difference learning models predict dopamine ramping. The key idea is representational: a quadratic transformation of proximity to the goal implies approximately linear ramping, as observed experimentally.
- 1 March 2014
- Published by MIT Press in Neural Computation
- Vol. 26 (3) , 467-471
Dopamine Ramps Are a Consequence of Reward Prediction Errors
Temporal difference learning models of dopamine assert that phasic levels of dopamine encode a reward prediction error. However, this hypothesis has been challenged by recent observations of gradually ramping stratal dopamine levels as a goal is approached. This note describes conditions under which temporal difference learning models predict dopamine ramping. The key idea is representational: a quadratic transformation of proximity to the goal implies approximately linear ramping, as observed experimentally.
- Open Access
- 1 January 2014
- Published by Frontiers Media SA in Frontiers in Computational Neuroscience
- Vol. 7, 194
Time representation in reinforcement learning models of the basal ganglia
Reinforcement learning models have been influential in understanding many aspects of basal ganglia function, from reward prediction to action selection. Time plays an important role in these models, but there is still no theoretical consensus about what kind of time representation is used by the basal ganglia. We review several theoretical accounts and their supporting evidence. We then discuss the relationship between reinforcement learning models and the timing mechanisms that have been attributed to the basal ganglia. We hypothesize that a single computational system may underlie both reinforcement learning and interval timing—the perception of duration in the range of seconds to hours. This hypothesis, which extends earlier models by incorporating a time-sensitive action selection mechanism, may have important implications for understanding disorders like Parkinson's disease in which both decision making and timing are impaired.
- 1 January 2014
- Published by Frontiers Media SA in Frontiers in Computational Neuroscience
- Vol. 7, 194
Open AccessTime representation in reinforcement learning models of the basal ganglia
Reinforcement learning models have been influential in understanding many aspects of basal ganglia function, from reward prediction to action selection. Time plays an important role in these models, but there is still no theoretical consensus about what kind of time representation is used by the basal ganglia. We review several theoretical accounts and their supporting evidence. We then discuss the relationship between reinforcement learning models and the timing mechanisms that have been attributed to the basal ganglia. We hypothesize that a single computational system may underlie both reinforcement learning and interval timing—the perception of duration in the range of seconds to hours. This hypothesis, which extends earlier models by incorporating a time-sensitive action selection mechanism, may have important implications for understanding disorders like Parkinson's disease in which both decision making and timing are impaired.