Ayaka Kato

The successor representation (SR) is a reinforcement learning strategy which has been suggested to underlie nuanced forms of habitual behavior, rendering it a candidate mechanism for mental health conditions thought to involve habitual control, such as addiction. However, paradigms used to assess habits in humans cannot detect SR use. Together with the fact that they are usually conducted in disorder-irrelevant contexts, this may explain why the evidence for a hypothesized shift from goal-directed to habitual behavior in addiction is inconsistent. Here, we tested whether individuals with high-risk (n=260) compared to low-risk drinking behavior (n=260) rely more on SR or a rigid goal-based dimension-reduced SR variant, particularly in alcohol-related contexts. An online multi-stage decision-making task performed in either a virtual bar or apartment context allowed us to dissociate (red)SR from model-based and model-free learning strategies by testing how well participants adapted their choices to different types of distal changes in the environment. Findings suggest that a particular type of SR, namely a (reduced) random-policy SR-like strategy, contributes to human behavior, but that high-risk drinkers do not differ from low-risk drinkers in their use of this strategy. Instead, both groups seem to rely less on what is likely a (reduced) random-policy SR strategy in alcohol-related contexts – and instead more on model-free control. Results suggest that the use of an adaptive, resource-efficient decision-making strategy is selectively downregulated in substance-related contexts. This study highlights the importance of considering contextual modulation and nuanced decision strategies in understanding habitual and goal-directed behavior in mental health conditions such as addiction.

Humans are remarkable at adapting to changing norms, yet we know little about how the human brain encodes these social signals at the single neuron level. Here, we recorded the activity of single neurons in the substantia nigra and globus pallidus in neurosurgical participants as they played an iterative ultimatum game versus human and computer avatars. Using computational modeling of behavior, we found that putative dopaminergic units in the substantia nigra tracked the valence of norm prediction errors, with greater encoding during gameplay with human versus computer avatars. Analogous analyses in the globus pallidus, a nucleus adjacent to the substantia nigra, found no significant effect. Our results identify a novel role for the substantia nigra in encoding social learning signals. AUTHOR SUMMARY: An important skill within social functioning is the ability to learn and adapt to changing social norms, or rules of behavior among social groups. While computational modeling of behavior has advanced our understanding of how humans learn social norms, little is known about what neural mechanisms support this process. Here, we leverage computational modeling of behavior in conjunction with human intracranial recording to investigate how the human brain encodes norm learning signals at the single neuron level. We recorded the activity of single neurons in the human substantia nigra pars reticulata (SNr) and globus pallidus internus (GPi) in neurosurgical participants as they played a social exchange game with human and computer avatars. We estimated trial-by-trial norm prediction errors (nPEs), or mentally computed errors in social prediction, and revealed that putative dopaminergic SNr neurons encoded the valence of nPEs, with greater encoding observed during gameplay with human versus computer avatar partners. This finding was not observed in the GPi. Together, our results expand upon existing knowledge about the SNr by identifying a novel role for SNr neurons in social norm learning.

Recent advances in deep brain stimulation (DBS) of the subcallosal cingulate (SCC) show promise in mitigating the symptoms of treatment-resistant depression (TRD) in humans ^1–3 . Monoamines, such as dopamine and serotonin, mediate the effects of pharmacological treatments of depression. However, their roles in recovery following DBS remain elusive, largely due to technical limitations of measuring these neurotransmitters in the living human brain. Here, by leveraging machine learning-enhanced electrochemistry ^4–7 , we show that dopamine and serotonin signaling following DBS to the SCC predicted later depressive symptom relief in humans with TRD. We found that both dopamine and serotonin levels increased following subtherapeutic intraoperative SCC stimulation, with each neurotransmitter showing selective responses to distinct decision-making tasks. Furthermore, acute dopamine increases predicted later mood improvements during a social decision-making task, while serotonin enhancement predicted faster responses during a non-social learning task longitudinally. Critically, changes in dopamine and serotonin levels during the social decision-making task jointly predicted depressive symptom remission at 6-month follow-up. These findings illustrate the contribution of both dopamine and serotonin signaling in predicting behavioral improvement and depressive symptom remission in humans with TRD. Such neurochemical plasticity may serve as potential mechanistic biomarkers for SCC DBS mechanism and TRD treatment response. Significance statement: Dopamine and serotonin levels increased following acute DBS to the SCC in humans. Acute dopamine and serotonin changes predicted later mood and response speed changes. Sustained TRD recovery was predicted by acute increases in both dopamine and serotonin estimates.

Machine learning approaches have advanced the identification of neural signatures of substance use, particularly through case-control comparisons and network-level analyses. However, most studies have focused on single substances in isolation, making it difficult to directly compare shared and distinct network computations and limiting their generalizability, given that many individuals engage in polysubstance use. Here, we introduce an explainable, connectivity-based multiclass classification framework that models distributed brain network organization to delineate both shared and substance-specific mechanisms of addiction. This design enables direct comparison among cannabis users, alcohol users, and controls, revealing network computations that uniquely differentiate each group. Using functional connectivity features from cue-induced craving tasks, we classified cannabis users (n=166), alcohol users (n=101), and healthy controls (n=238), achieving high out-of-sample accuracy (87% for cannabis, 69% for alcohol, and 73% for controls). Functional connectivity-based models consistently outperformed activation-based models, highlighting the importance of inter-regional network properties for biomarker development. Network analysis further revealed dorsal prefrontal, cingulate, and precuneus hubs consistent with meta-analytic craving networks and substance-specific connectivity motifs enhanced prefrontal coupling in cannabis users and greater insular-striatal integration in alcohol users. These results demonstrate that distinct network configurations define different substance-use profiles, advancing interpretable biomarkers for addiction neuroscience.

Representation of external and internal states in the brain plays a critical role in enabling suitable behavior. Recent studies suggest that state representation and state value can be simultaneously learned through Temporal-Difference-Reinforcement-Learning (TDRL) and Backpropagation-Through-Time (BPTT) in recurrent neural networks (RNNs) and their readout. However, neural implementation of such learning remains unclear as BPTT requires offline update using transported downstream weights, which is suggested to be biologically implausible. We demonstrate that simple online training of RNNs using TD reward prediction error and random feedback, without additional memory or eligibility trace, can still learn the structure of tasks with cue–reward delay and timing variability. This is because TD learning itself is a solution for temporal credit assignment, and feedback alignment, a mechanism originally proposed for supervised learning, enables gradient approximation without weight transport. Furthermore, we show that biologically constraining downstream weights and random feedback to be non-negative not only preserves learning but may even enhance it because the non-negative constraint ensures loose alignment—allowing the downstream and feedback weights to roughly align from the beginning. These results provide insights into the neural mechanisms underlying the learning of state representation and value, highlighting the potential of random feedback and biological constraints.

Representation of external and internal states in the brain plays a critical role in enabling suitable behavior. Recent studies suggest that state representation and state value can be simultaneously learnt through Temporal-Difference-Reinforcement-Learning (TDRL) and Backpropagation-Through-Time (BPTT) in recurrent neural networks (RNNs) and their readout. However, neural implementation of such learning remains unclear as BPTT requires offline update using transported downstream weights, which is suggested to be biologically implausible. We demonstrate that simple online training of RNNs using TD reward prediction error and random feedback, without additional memory or eligibility trace, can still learn the structure of tasks with cue-reward delay and timing variability. This is because TD learning itself is a solution for temporal credit assignment, and feedback alignment, a mechanism originally proposed for supervised learning, enables gradient approximation without weight transport. Furthermore, we show that biologically constraining downstream weights and random feedback to be non-negative not only preserves learning but may even enhance it because the non-negative constraint ensures loose alignment - allowing the downstream and feedback weights to roughly align from the beginning. These results provide insights into the neural mechanisms underlying the learning of state representation and value, highlighting the potential of random feedback and biological constraints.

Dopamine has been suggested to encode reward-prediction-error (RPE) in reinforcement learning (RL) theory but also shown to exhibit heterogeneous patterns depending on regions and conditions: some exhibiting ramping response to predictable reward while others only responding to reward-predicting cue. It remains elusive how these heterogeneities relate to various RL algorithms proposed to be employed by animals/humans, such as RL under predictive state representation, hierarchical RL, and distributional RL. Here we demonstrate that these relationships can be coherently explained by incorporating the decay of learned values (value-decay), implementable by the decay of dopamine-dependent plastic changes in the synaptic strengths. First, we show that value-decay causes ramping RPE under certain state representations but not under others. This accounted for the observed gradual fading of dopamine ramping across repeated reward navigation, attributed to the gradual formation of predictive state representations. It also explained the cue-type and inter-trial-interval-dependent temporal patterns of dopamine. Next, we constructed a hierarchical RL model composed of two coupled systems—one with value-decay and one without. The model accounted for distinct patterns of neuronal activity in parallel striatal-dopamine circuits and their proposed roles in flexible learning and stable habit formation. Lastly, we examined two distinct algorithms of distributional RL with and without value-decay. These algorithms explained how distinct dopamine patterns across striatal regions relate to the reported differences in the strength of distributional coding. These results suggest that within-striatum differences—specifically, a medial-to-lateral gradient in value or synaptic decay—tune regional RL computations by generating distinct patterns of dopamine/RPE signals.

Dopamine has been suggested to encode reward-prediction-error (RPE) in reinforcement learning (RL) theory, but also shown to exhibit heterogeneous patterns depending on regions and conditions: some exhibiting ramping response to predictable reward while others only responding to reward-predicting cue. It remains elusive how these heterogeneities relate to various RL algorithms proposed to be employed by animals/humans, such as RL under predictive state representation, hierarchical RL, and distributional RL. Here we demonstrate that these relationships can be coherently explained by incorporating the decay of learned values (value-decay), implementable by the decay of dopamine-dependent plastic changes in the synaptic strengths. First, we show that value-decay causes ramping RPE under certain state representations but not under others. This accounted for the observed gradual fading of dopamine ramping across repeated reward navigation, attributed to the gradual formation of predictive state representations. It also explained the cue-type and inter-trial-interval-dependent temporal patterns of dopamine. Next, we constructed a hierarchical RL model composed of two coupled systems—one with value-decay and one without. The model accounted for distinct patterns of neuronal activity in parallel striatal-dopamine circuits and their proposed roles in flexible learning and stable habit formation. Lastly, we examined two distinct algorithms of distributional RL with and without value-decay. These algorithms explained how distinct dopamine patterns across striatal regions relate to the reported differences in the strength of distributional coding. These results suggest that within-striatum differences—specifically, a medial-to-lateral gradient in value or synaptic decay—tune regional RL computations by generating distinct patterns of dopamine/RPE signals.

Computational models have become integral to human neuroimaging research, providing both mechanistic insights and predictive tools for human cognition and behavior. However, concerns persist regarding the ecological validity of lab-based neuroimaging studies and whether their spatiotemporal resolution is not sufficient for capturing neural dynamics. This review aims to re-examine the utility of computational neuroimaging, particularly in light of the growing prominence of alternative neuroscientific methods and the growing emphasis on more naturalistic behaviors and paradigms. Specifically, we will explore how computational modeling can both enhance the analysis of high-dimensional imaging datasets and, conversely, how neuroimaging, in conjunction with other data modalities, can inform computational models through the lens of neurobiological plausibility. Collectively, this evidence suggests that neuroimaging remains critical for human neuroscience research, and when enhanced by computational models, imaging can serve an important role in bridging levels of analysis and understanding. We conclude by proposing key directions for future research, emphasizing the development of standardized paradigms and the integrative use of computational modeling across neuroimaging techniques.

Representation of external and internal states in the brain plays a critical role in enabling suitable behavior. Recent studies suggest that state representation and state value can be simultaneously learnt through Temporal-Difference-Reinforcement-Learning (TDRL) and Backpropagation-Through-Time (BPTT) in recurrent neural networks (RNNs) and their readout. However, neural implementation of such learning remains unclear as BPTT requires offline update using transported downstream weights, which is suggested to be biologically implausible. We demonstrate that simple online training of RNNs using TD reward prediction error and random feedback, without additional memory or eligibility trace, can still learn the structure of tasks with cue-reward delay and timing variability. This is because TD learning itself is a solution for temporal credit assignment, and feedback alignment, a mechanism originally proposed for supervised learning, enables gradient approximation without weight transport. Furthermore, we show that biologically constraining downstream weights and random feedback to be non-negative not only preserves learning but may even enhance it because the non-negative constraint ensures loose alignment - allowing the downstream and feedback weights to roughly align from the beginning. These results provide insights into the neural mechanisms underlying the learning of state representation and value, highlighting the potential of random feedback and biological constraints.