Kanji Shimomura
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Kanji Shimomura has a total of 21 co-authors. Here are the top co-authors by number of shared publications:
Kenji Morita
7 shared publications- Kenji Morita
Physical and Health Education, Graduate School of Education, The University of Tokyo
4 shared publications - Ayaka Kato
Department of Psychiatry, Icahn School of Medicine at Mount Sinai
3 shared publications Jeff C. Huffman
Department of Psychiatry, Massachusetts General Hospital, Boston, USA
2 shared publicationsRachel A. Millstein
Department of Psychiatry, Massachusetts General Hospital, Boston, USA
2 shared publicationsYuki Nishiguchi
Faculty of Education, Chiba University, Chiba, Japan
2 shared publications
Latest Publications
- Open Access
- 13 October 2026
- Published by Center for Open Science in PsyArXiv
The successor representation in high-risk drinking and alcohol-related contexts
The successor representation (SR) is a reinforcement learning strategy which has been suggested to underlie nuanced forms of habitual behavior, rendering it a candidate mechanism for mental health conditions thought to involve habitual control, such as addiction. However, paradigms used to assess habits in humans cannot detect SR use. Together with the fact that they are usually conducted in disorder-irrelevant contexts, this may explain why the evidence for a hypothesized shift from goal-directed to habitual behavior in addiction is inconsistent. Here, we tested whether individuals with high-risk (n=260) compared to low-risk drinking behavior (n=260) rely more on SR or a rigid goal-based dimension-reduced SR variant, particularly in alcohol-related contexts. An online multi-stage decision-making task performed in either a virtual bar or apartment context allowed us to dissociate (red)SR from model-based and model-free learning strategies by testing how well participants adapted their choices to different types of distal changes in the environment. Findings suggest that a particular type of SR, namely a (reduced) random-policy SR-like strategy, contributes to human behavior, but that high-risk drinkers do not differ from low-risk drinkers in their use of this strategy. Instead, both groups seem to rely less on what is likely a (reduced) random-policy SR strategy in alcohol-related contexts – and instead more on model-free control. Results suggest that the use of an adaptive, resource-efficient decision-making strategy is selectively downregulated in substance-related contexts. This study highlights the importance of considering contextual modulation and nuanced decision strategies in understanding habitual and goal-directed behavior in mental health conditions such as addiction.
- 13 October 2026
- Published by Center for Open Science in PsyArXiv
Open AccessThe successor representation in high-risk drinking and alcohol-related contexts
The successor representation (SR) is a reinforcement learning strategy which has been suggested to underlie nuanced forms of habitual behavior, rendering it a candidate mechanism for mental health conditions thought to involve habitual control, such as addiction. However, paradigms used to assess habits in humans cannot detect SR use. Together with the fact that they are usually conducted in disorder-irrelevant contexts, this may explain why the evidence for a hypothesized shift from goal-directed to habitual behavior in addiction is inconsistent. Here, we tested whether individuals with high-risk (n=260) compared to low-risk drinking behavior (n=260) rely more on SR or a rigid goal-based dimension-reduced SR variant, particularly in alcohol-related contexts. An online multi-stage decision-making task performed in either a virtual bar or apartment context allowed us to dissociate (red)SR from model-based and model-free learning strategies by testing how well participants adapted their choices to different types of distal changes in the environment. Findings suggest that a particular type of SR, namely a (reduced) random-policy SR-like strategy, contributes to human behavior, but that high-risk drinkers do not differ from low-risk drinkers in their use of this strategy. Instead, both groups seem to rely less on what is likely a (reduced) random-policy SR strategy in alcohol-related contexts – and instead more on model-free control. Results suggest that the use of an adaptive, resource-efficient decision-making strategy is selectively downregulated in substance-related contexts. This study highlights the importance of considering contextual modulation and nuanced decision strategies in understanding habitual and goal-directed behavior in mental health conditions such as addiction.
- Open Access
- 8 December 2025
- Published by Taylor & Francis in European Journal of Psychotraumatology
- Vol. 16 (1) , 2592408
Long-term mental health trajectories after the Great East Japan Earthquake: a systematic review
Background: The Great East Japan Earthquake (GEJE) of 2011, compounded by the ensuing tsunami and Fukushima nuclear disaster, had profound impacts on the affected populations. While the immediate psychological consequences of the disaster have been well-documented, less is known about the long-term mental health trajectories of survivors.Objective: This systematic review aims to synthesize existing longitudinal studies to provide a comprehensive overview of long-term mental health outcomes following the GEJE.Methods: This review was conducted in accordance with PRISMA guidelines. Using predefined inclusion criteria, a comprehensive search across PubMed, Web of Science, APA PsycINFO, and Ichushi was performed from July 2024 to May 2025. We examined studies assessing post-traumatic stress symptoms (PTSS), depression, and psychological distress among GEJE survivors with multi-wave follow-up data where at least one assessment took place 1 year or longer after the disaster.Results: We screened 1,719 references and identified 30 reports of 27 longitudinal studies encompassing over 197,351 participants. The included studies varied in sample characteristics, measurement tools, cut-off criteria, and follow-up time points, making direct comparisons challenging. Across most studies, outcomes for PTSS, depression, and psychological distress showed an initial peak in symptoms following the disaster, which gradually declined over time. In some studies, PTSS appeared to show a sharper decline compared to depression or psychological distress. Variations were observed among different subgroups, including long-term evacuees, women, and older individuals, who tended to exhibit higher or more sustained levels of symptoms.Conclusions: This systematic review highlights that the GEJE has had enduring impacts on survivors' mental health. While PTSS tends to decline substantially over time, psychological distress and depression also decrease, albeit in a slower manner. Long-term mental health care should be provided, especially for vulnerable subgroups, to mitigate the disaster's lasting psychological impacts.
- 8 December 2025
- Published by Taylor & Francis in European Journal of Psychotraumatology
- Vol. 16 (1) , 2592408
Open AccessLong-term mental health trajectories after the Great East Japan Earthquake: a systematic review
Background: The Great East Japan Earthquake (GEJE) of 2011, compounded by the ensuing tsunami and Fukushima nuclear disaster, had profound impacts on the affected populations. While the immediate psychological consequences of the disaster have been well-documented, less is known about the long-term mental health trajectories of survivors.Objective: This systematic review aims to synthesize existing longitudinal studies to provide a comprehensive overview of long-term mental health outcomes following the GEJE.Methods: This review was conducted in accordance with PRISMA guidelines. Using predefined inclusion criteria, a comprehensive search across PubMed, Web of Science, APA PsycINFO, and Ichushi was performed from July 2024 to May 2025. We examined studies assessing post-traumatic stress symptoms (PTSS), depression, and psychological distress among GEJE survivors with multi-wave follow-up data where at least one assessment took place 1 year or longer after the disaster.Results: We screened 1,719 references and identified 30 reports of 27 longitudinal studies encompassing over 197,351 participants. The included studies varied in sample characteristics, measurement tools, cut-off criteria, and follow-up time points, making direct comparisons challenging. Across most studies, outcomes for PTSS, depression, and psychological distress showed an initial peak in symptoms following the disaster, which gradually declined over time. In some studies, PTSS appeared to show a sharper decline compared to depression or psychological distress. Variations were observed among different subgroups, including long-term evacuees, women, and older individuals, who tended to exhibit higher or more sustained levels of symptoms.Conclusions: This systematic review highlights that the GEJE has had enduring impacts on survivors' mental health. While PTSS tends to decline substantially over time, psychological distress and depression also decrease, albeit in a slower manner. Long-term mental health care should be provided, especially for vulnerable subgroups, to mitigate the disaster's lasting psychological impacts.
- Open Access
- 11 August 2025
- Published by Springer Nature in Computational Brain & Behavior
- Vol. 9 (1) , 133-153
State Optimism as a State-Dependent Prior About Rewarding Outcomes
Optimism has recently been shown to have a state-like aspect that fluctuates within individuals. Given that optimism represents expectations for one’s own future, its intraindividual changes may interact with learning and decision-making. However, how optimism can be placed in a formal learning theory remains elusive. Here, we tested potential dynamic relationships between state optimism and learning by conducting behavioral tasks and questionnaire surveys pre- and post-intervention to increase optimism ( n = 95 for the intervention group and n = 41 for the control group), which has been shown to be effective for trait optimism in previous meta-analyses. After demonstrating that this intervention was also effective for state optimism, the following preregistered hypotheses were tested: intraindividual changes in state optimism (1) relate to positive belief updating bias and (2) relate to prior expectation about reward likelihood. While the results did not support hypothesis (1), hypothesis (2) was partially supported. Specifically, intraindividual changes in prior about reward, estimated through a reward learning task, showed a positive association with changes in state optimism, measured by questionnaire, in the intervention group. This result suggests a functional role of state optimism in value-based learning and decision-making: it defines a state-dependent general prior of rewarding outcomes, which can optimize the degree of exploration under that state.
- 11 August 2025
- Published by Springer Nature in Computational Brain & Behavior
- Vol. 9 (1) , 133-153
Open AccessState Optimism as a State-Dependent Prior About Rewarding Outcomes
Optimism has recently been shown to have a state-like aspect that fluctuates within individuals. Given that optimism represents expectations for one’s own future, its intraindividual changes may interact with learning and decision-making. However, how optimism can be placed in a formal learning theory remains elusive. Here, we tested potential dynamic relationships between state optimism and learning by conducting behavioral tasks and questionnaire surveys pre- and post-intervention to increase optimism ( n = 95 for the intervention group and n = 41 for the control group), which has been shown to be effective for trait optimism in previous meta-analyses. After demonstrating that this intervention was also effective for state optimism, the following preregistered hypotheses were tested: intraindividual changes in state optimism (1) relate to positive belief updating bias and (2) relate to prior expectation about reward likelihood. While the results did not support hypothesis (1), hypothesis (2) was partially supported. Specifically, intraindividual changes in prior about reward, estimated through a reward learning task, showed a positive association with changes in state optimism, measured by questionnaire, in the intervention group. This result suggests a functional role of state optimism in value-based learning and decision-making: it defines a state-dependent general prior of rewarding outcomes, which can optimize the degree of exploration under that state.
- Open Access
- 13 November 2024
- Published by Center for Open Science in PsyArXiv
State optimism as a state-dependent prior about rewarding outcome
Optimism has recently been shown to have a state-like aspect that fluctuates within individuals. Given that optimism represents expectations for one’s own future, its intraindividual changes may interact with learning and decision-making. Using an intervention to increase optimism (n = 95 for the intervention group and n = 41 for the control group), this preregistered study tested the potential dynamic interaction between state optimism and two aspects of learning: positive belief updating bias and prior about reward likelihood. Although intraindividual changes in state optimism did not show significant association with updating bias, they showed a positive association with changes in prior about reward in the intervention group. This result offers a novel insight into a functional role of state optimism in reward learning: it may offer a state-dependent general prior of rewarding outcomes, which is applied to arbitrary learning occasions encountered during that state.
- 13 November 2024
- Published by Center for Open Science in PsyArXiv
Open AccessState optimism as a state-dependent prior about rewarding outcome
Optimism has recently been shown to have a state-like aspect that fluctuates within individuals. Given that optimism represents expectations for one’s own future, its intraindividual changes may interact with learning and decision-making. Using an intervention to increase optimism (n = 95 for the intervention group and n = 41 for the control group), this preregistered study tested the potential dynamic interaction between state optimism and two aspects of learning: positive belief updating bias and prior about reward likelihood. Although intraindividual changes in state optimism did not show significant association with updating bias, they showed a positive association with changes in prior about reward in the intervention group. This result offers a novel insight into a functional role of state optimism in reward learning: it may offer a state-dependent general prior of rewarding outcomes, which is applied to arbitrary learning occasions encountered during that state.
- Open Access
- 3 October 2024
- Published by Springer Nature in Computational Brain & Behavior
- Vol. 8 (1) , 147-161
The Effect of Reward Magnitude on Different Types of Exploration in Human Reinforcement Learning
How humans resolve the explore–exploit dilemma in complex environments is an important open question. Previous studies suggested that environmental richness may affect the degree of exploration in a type-specific manner and reduce random exploration while increasing uncertainty-based exploration. Our study examined this possibility by extending a recently developed two-armed bandit task that can dissociate the uncertainty and novelty of stimuli. To extract the pure effect of environmental richness, we manipulated the reward by its magnitude, not its probability, across blocks because reward probability affects outcome controllability. Participants (N = 198) demonstrated increased optimal choices when the relative reward magnitude was higher. A behavioral analysis with computational modeling revealed that a higher reward magnitude reduced the degree of random exploration but had little effect on the degree of uncertainty- and novelty-based exploration. These results suggest that humans modulate their degree of random exploration depending on the relative level of environmental richness. Combined with findings from previous studies, our findings indicate the possibility that outcome controllability also influences the exploration–exploitation balance in human reinforcement learning.
- 3 October 2024
- Published by Springer Nature in Computational Brain & Behavior
- Vol. 8 (1) , 147-161
Open AccessThe Effect of Reward Magnitude on Different Types of Exploration in Human Reinforcement Learning
How humans resolve the explore–exploit dilemma in complex environments is an important open question. Previous studies suggested that environmental richness may affect the degree of exploration in a type-specific manner and reduce random exploration while increasing uncertainty-based exploration. Our study examined this possibility by extending a recently developed two-armed bandit task that can dissociate the uncertainty and novelty of stimuli. To extract the pure effect of environmental richness, we manipulated the reward by its magnitude, not its probability, across blocks because reward probability affects outcome controllability. Participants (N = 198) demonstrated increased optimal choices when the relative reward magnitude was higher. A behavioral analysis with computational modeling revealed that a higher reward magnitude reduced the degree of random exploration but had little effect on the degree of uncertainty- and novelty-based exploration. These results suggest that humans modulate their degree of random exploration depending on the relative level of environmental richness. Combined with findings from previous studies, our findings indicate the possibility that outcome controllability also influences the exploration–exploitation balance in human reinforcement learning.
- Open Access
- 1 January 2024
- Published by The Japanese Psychological Association in The Proceedings of the Annual Convention of the Japanese Psychological Association
- Vol. 88, 1B-085
The relationship between intraindividual change in optimism and biases in learning
- 1 January 2024
- Published by The Japanese Psychological Association in The Proceedings of the Annual Convention of the Japanese Psychological Association
- Vol. 88, 1B-085
Open AccessThe relationship between intraindividual change in optimism and biases in learning
- Open Access
- 20 November 2023
- Published by Springer Nature in Affective Science
- Vol. 5 (2) , 1-12
Intraindividual Fluctuation in Optimism Under Daily Life Circumstances: A Longitudinal Study
Optimism is typically conceptualized as a relatively static tendency regarding positive expectations about one’s future. However, recent studies suggest that optimism may meaningfully fluctuate within individuals over time. To date, little is known about the characteristics of such state optimism and potential cultural difference in state optimism. Accordingly, we developed a Japanese version of the State Optimism Measure (J-SOM) and examined its validity and the nature of intraindividual state optimism fluctuations; we also examined relationships between the J-SOM and other measures of mental health, including trait optimism. We conducted two online longitudinal surveys with different time intervals (weekly, n = 97; monthly, n = 99) targeting university students. Results were largely consistent between the two surveys. We confirmed high factor validity and internal consistency of the J-SOM. The J-SOM showed significant correlations in expected directions with other measures such as depressive mood and subjective happiness. In addition, intraindividual changes in the J-SOM were associated with changes in mood and quality of daily life. Importantly, these associations between intraindividual change in optimism and in other variables were minimal for trait optimism. We also found that state optimism, compared with trait optimism, tended to show larger intraindividual changes over 1, 2, 3, 4, and 8 weeks. In summary, this study developed a translated version of the SOM and validated it, and then showed, for the first time, that state optimism can fluctuate within individuals in daily life over a span of several weeks.
- 20 November 2023
- Published by Springer Nature in Affective Science
- Vol. 5 (2) , 1-12
Open AccessIntraindividual Fluctuation in Optimism Under Daily Life Circumstances: A Longitudinal Study
Optimism is typically conceptualized as a relatively static tendency regarding positive expectations about one’s future. However, recent studies suggest that optimism may meaningfully fluctuate within individuals over time. To date, little is known about the characteristics of such state optimism and potential cultural difference in state optimism. Accordingly, we developed a Japanese version of the State Optimism Measure (J-SOM) and examined its validity and the nature of intraindividual state optimism fluctuations; we also examined relationships between the J-SOM and other measures of mental health, including trait optimism. We conducted two online longitudinal surveys with different time intervals (weekly, n = 97; monthly, n = 99) targeting university students. Results were largely consistent between the two surveys. We confirmed high factor validity and internal consistency of the J-SOM. The J-SOM showed significant correlations in expected directions with other measures such as depressive mood and subjective happiness. In addition, intraindividual changes in the J-SOM were associated with changes in mood and quality of daily life. Importantly, these associations between intraindividual change in optimism and in other variables were minimal for trait optimism. We also found that state optimism, compared with trait optimism, tended to show larger intraindividual changes over 1, 2, 3, 4, and 8 weeks. In summary, this study developed a translated version of the SOM and validated it, and then showed, for the first time, that state optimism can fluctuate within individuals in daily life over a span of several weeks.
- Open Access
- 20 January 2023
- Published by Center for Open Science in PsyArXiv
Intraindividual fluctuation in optimism: Development and validation of the Japanese version of the State Optimism Measure
This study aimed to develop and validate the Japanese version of the State Optimism Measure (J-SOM) and to examine the nature of the intraindividual fluctuation of state optimism. We conducted two online longitudinal surveys (n=97 and n=99) targeting university students. The results indicated the high factor validity and internal consistency of J-SOM. The J-SOM showed significant correlations in the expected direction with other measures. In addition, the intraindividual change in the J-SOM associated with the change in mood and the quality of daily life, while such association was minimal for trait optimism. We also found that state optimism, compared with trait optimism, showed larger intraindividual change in the intervals of two and three weeks. In summary, this study not only demonstrated the validity of newly developed J-SOM, but also indicated that state optimism can fluctuate within individuals under daily life over a span of several weeks.
- 20 January 2023
- Published by Center for Open Science in PsyArXiv
Open AccessIntraindividual fluctuation in optimism: Development and validation of the Japanese version of the State Optimism Measure
This study aimed to develop and validate the Japanese version of the State Optimism Measure (J-SOM) and to examine the nature of the intraindividual fluctuation of state optimism. We conducted two online longitudinal surveys (n=97 and n=99) targeting university students. The results indicated the high factor validity and internal consistency of J-SOM. The J-SOM showed significant correlations in the expected direction with other measures. In addition, the intraindividual change in the J-SOM associated with the change in mood and the quality of daily life, while such association was minimal for trait optimism. We also found that state optimism, compared with trait optimism, showed larger intraindividual change in the intervals of two and three weeks. In summary, this study not only demonstrated the validity of newly developed J-SOM, but also indicated that state optimism can fluctuate within individuals under daily life over a span of several weeks.
- Open Access
Opponent Learning with Different Representations in the Cortico-Basal Ganglia Pathways Can Develop Obsession-Compulsion Cycle
Obsessive-compulsive disorder (OCD) has been suggested to be associated with impairment of model-based behavioral control. Meanwhile, recent work suggested shorter memory trace for negative than positive prediction errors (PEs) in OCD. We explored relations between these two suggestions through computational modeling. Based on the properties of cortico-basal ganglia pathways, we modeled human as an agent having a combination of successor representation (SR)-based system that enables model-based-like control and individual representation (IR)-based system that only hosts model-free control, with the two systems potentially learning from positive and negative PEs in different rates. We simulated the agent’s behavior in the environmental model used in the recent work that describes potential development of obsession-compulsion cycle. We found that the dual-system agent could develop enhanced obsession-compulsion cycle, similarly to the agent having memory trace imbalance in the recent work, if the SR- and IR-based systems learned mainly from positive and negative PEs, respectively. We then simulated the behavior of such an opponent SR+IR agent in the two-stage decision task, in comparison with the agent having only SR-based control. Fitting of the agents’ behavior by the model weighing model-based and model-free control developed in the original two-stage task study resulted in smaller weights of model-based control for the opponent SR+IR agent than for the SR-only agent. These results reconcile the previous suggestions about OCD, i.e., impaired model-based control and memory trace imbalance, raising a novel possibility that opponent learning in model(SR)-based and model-free controllers underlies obsession-compulsion. Our model cannot explain the behavior of OCD patients in punishment, rather than reward, contexts, but it could be resolved if opponent SR+IR learning operates also in the recently revealed non-canonical cortico-basal ganglia-dopamine circuit for threat/aversiveness, rather than reward, reinforcement learning, and the aversive SR + appetitive IR agent could actually develop obsession-compulsion if the environment is modeled differently. Author summary: Obsessive-compulsive disorder (OCD) is one of the major psychiatric disorders diagnosed in 2.5%-3% of the population, and is characterized as an enhanced cycle of obsessive thought, e.g., whether the door was locked, and compulsive action, e.g., checking door lock. It remains elusive why such an apparently maladaptive behavior could be enhanced. A prevailing theory proposes that humans use two control systems, flexible yet costly goal-directed system and inflexible yet costless habitual system, and impairment of the goal-directed system leads to OCD. On the other hand, recent work proposed a new theory that shorter memory trace for credit-assignment of negative, than positive, prediction errors can induce OCD. Relation between these two theories remains unclear. We show that opponent learning of particular type of goal-directed(-like) system, suggested to be implemented in the brain, and habitual system from positive versus negative prediction errors could exhibit an (apparent) overall decrease in goal-directed control and also develop enhanced obsession-compulsion cycle similar to the one developed by memory-trace imbalance, thereby bridging the two theories. Such an opponent learning of the two systems was actually suggested to be advantageous in certain dynamic environments, and could thus be evolutionarily selected at the cost of possible development of OCD.
Open AccessOpponent Learning with Different Representations in the Cortico-Basal Ganglia Pathways Can Develop Obsession-Compulsion Cycle
Obsessive-compulsive disorder (OCD) has been suggested to be associated with impairment of model-based behavioral control. Meanwhile, recent work suggested shorter memory trace for negative than positive prediction errors (PEs) in OCD. We explored relations between these two suggestions through computational modeling. Based on the properties of cortico-basal ganglia pathways, we modeled human as an agent having a combination of successor representation (SR)-based system that enables model-based-like control and individual representation (IR)-based system that only hosts model-free control, with the two systems potentially learning from positive and negative PEs in different rates. We simulated the agent’s behavior in the environmental model used in the recent work that describes potential development of obsession-compulsion cycle. We found that the dual-system agent could develop enhanced obsession-compulsion cycle, similarly to the agent having memory trace imbalance in the recent work, if the SR- and IR-based systems learned mainly from positive and negative PEs, respectively. We then simulated the behavior of such an opponent SR+IR agent in the two-stage decision task, in comparison with the agent having only SR-based control. Fitting of the agents’ behavior by the model weighing model-based and model-free control developed in the original two-stage task study resulted in smaller weights of model-based control for the opponent SR+IR agent than for the SR-only agent. These results reconcile the previous suggestions about OCD, i.e., impaired model-based control and memory trace imbalance, raising a novel possibility that opponent learning in model(SR)-based and model-free controllers underlies obsession-compulsion. Our model cannot explain the behavior of OCD patients in punishment, rather than reward, contexts, but it could be resolved if opponent SR+IR learning operates also in the recently revealed non-canonical cortico-basal ganglia-dopamine circuit for threat/aversiveness, rather than reward, reinforcement learning, and the aversive SR + appetitive IR agent could actually develop obsession-compulsion if the environment is modeled differently. Author summary: Obsessive-compulsive disorder (OCD) is one of the major psychiatric disorders diagnosed in 2.5%-3% of the population, and is characterized as an enhanced cycle of obsessive thought, e.g., whether the door was locked, and compulsive action, e.g., checking door lock. It remains elusive why such an apparently maladaptive behavior could be enhanced. A prevailing theory proposes that humans use two control systems, flexible yet costly goal-directed system and inflexible yet costless habitual system, and impairment of the goal-directed system leads to OCD. On the other hand, recent work proposed a new theory that shorter memory trace for credit-assignment of negative, than positive, prediction errors can induce OCD. Relation between these two theories remains unclear. We show that opponent learning of particular type of goal-directed(-like) system, suggested to be implemented in the brain, and habitual system from positive versus negative prediction errors could exhibit an (apparent) overall decrease in goal-directed control and also develop enhanced obsession-compulsion cycle similar to the one developed by memory-trace imbalance, thereby bridging the two theories. Such an opponent learning of the two systems was actually suggested to be advantageous in certain dynamic environments, and could thus be evolutionarily selected at the cost of possible development of OCD.
- Open Access
Opponent learning with different representations in the cortico-basal ganglia circuits
The direct and indirect pathways of the basal ganglia (BG) have been suggested to learn mainly from positive and negative feedbacks, respectively. Since these pathways unevenly receive inputs from different cortical neuron types and/or regions, they may preferentially use different state/action representations. We explored whether such combined use of different representations coupled with appetitive or aversive learning has computational benefits. We simulated reward learning tasks in dynamic environments, and examined the performance of animal modeled as an agent equipped with two learning systems, each of which used individual representation (IR) or successor representation (SR) of states. With varying the combination of IR or SR and also the learning rates from positive and negative reward prediction errors (RPEs) in each system, we found that combination of an SR-based system learning mainly from positive RPEs and an IR-based system learning mainly from negative RPEs could achieve good performance, as compared to other combinations, in many situations. The architecture of such a combination provides a novel coherent explanation for the functional significance and underlying mechanism of diverse findings about the cortico-BG circuits. These results suggest that combining different representations with appetitive and aversive learning is an effective learning strategy adopted by the brain.
Open AccessOpponent learning with different representations in the cortico-basal ganglia circuits
The direct and indirect pathways of the basal ganglia (BG) have been suggested to learn mainly from positive and negative feedbacks, respectively. Since these pathways unevenly receive inputs from different cortical neuron types and/or regions, they may preferentially use different state/action representations. We explored whether such combined use of different representations coupled with appetitive or aversive learning has computational benefits. We simulated reward learning tasks in dynamic environments, and examined the performance of animal modeled as an agent equipped with two learning systems, each of which used individual representation (IR) or successor representation (SR) of states. With varying the combination of IR or SR and also the learning rates from positive and negative reward prediction errors (RPEs) in each system, we found that combination of an SR-based system learning mainly from positive RPEs and an IR-based system learning mainly from negative RPEs could achieve good performance, as compared to other combinations, in many situations. The architecture of such a combination provides a novel coherent explanation for the functional significance and underlying mechanism of diverse findings about the cortico-BG circuits. These results suggest that combining different representations with appetitive and aversive learning is an effective learning strategy adopted by the brain.