Alain R. Marchand

Adaptive decision-making in dynamic environments requires flexible adjustment of learning speed to balance stability and flexibility. When outcomes are highly stochastic, learners must avoid over interpreting noise and update more slowly, whereas in volatile environments where contingencies change frequently, learning should accelerate to rapidly incorporate new evidence. Theories propose that internal estimates of uncertainty tune learning rates through neuromodulatory-dependent mechanisms. Here, we investigated how noradrenergic inputs from the locus coeruleus (LC) to the orbitofrontal cortex (OFC) support adaptive learning under uncertainty. We show that rats performing a probabilistic reversal learning task exhibited behavior that was best explained by an adaptive reinforcement-learning model in which learning rates dynamically adjust according to estimated environmental volatility and stochasticity, outperforming standard fixed-rate models. Noradrenaline release in the OFC closely tracked trial-by-trial, model-derived volatility estimates around contingency changes. Disrupting LC→OFC noradrenergic inputs reproduced the model-predicted deficit in volatility-dependent adjustments of learning rate. Together, these findings identify OFC noradrenergic signaling as a key circuit mechanism for volatility-dependent modulation of learning rates during adaptive decision-making.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade‐off has been extensively studied in pretrained animals facing non‐stationary decision‐making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long‐term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three‐armed bandit task across 24 days of pretraining. We found that the day‐by‐day evolution of rat performance and win‐shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta‐learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

In the context of Pavlovian conditioning, two types of behaviour may emerge within the population (Flagel et al. Nature, 469(7328): 53–57, 2011 ). Animals may choose to engage either with the conditioned stimulus (CS), a behaviour known as sign-tracking (ST) which is sensitive to dopamine inhibition for its acquisition, or with the food cup in which the reward or unconditioned stimulus (US) will eventually be delivered, a behaviour known as goal-tracking (GT) which is dependent on dopamine for its expression only. Previous work by Lesaint et al. (PLoS Comput Biol, 10(2), 2014 ) offered a computational explanation for these phenomena and led to the prediction that varying the duration of the inter-trial interval (ITI) would change the relative ST-GT proportion in the population as well as phasic dopamine responses. A recent study verified this prediction, but also found a rich variance of ST and GT behaviours within the trial which goes beyond the original computational model. In this paper, we provide a computational perspective on these novel results.

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies.

Recent computational models of sign tracking (ST) and goal tracking (GT) have accounted for observations that dopamine (DA) is not necessary for all forms of learning and have provided a set of predictions to further their validity. Among these, a central prediction is that manipulating the intertrial interval (ITI) during autoshaping should change the relative ST-GT proportion as well as DA phasic responses. Here, we tested these predictions and found that lengthening the ITI increased ST, i.e., behavioral engagement with conditioned stimuli (CS) and cue-induced phasic DA release. Importantly, DA release was also present at the time of reward delivery, even after learning, and DA release was correlated with time spent in the food cup during the ITI. During conditioning with shorter ITIs, GT was prominent (i.e., engagement with food cup), and DA release responded to the CS while being absent at the time of reward delivery after learning. Hence, shorter ITIs restored the classical DA reward prediction error (RPE) pattern. These results validate the computational hypotheses, opening new perspectives on the understanding of individual differences in Pavlovian conditioning and DA signaling. In classical or Pavlovian conditioning, subjects learn to associate a previously neutral stimulus (called “conditioned” stimulus; for example, a bell) with a biologically potent stimulus (called “unconditioned” stimulus; for example, a food reward). In some animals, the incentive salience of the conditioned stimuli is so strong that the conditioned response is to engage the conditioned stimuli instead of immediately approaching the food cup, where the predicted food will be delivered. These animals are referred to as “sign trackers.” Other animals, referred to as “goal trackers,” proceed directly to the food cup upon presentation of the conditioned stimulus to obtain reward. Understanding the mechanisms by which these divergent behaviors develop under identical environmental conditions will provide powerful insight into the neurobiological substrates underlying learning. Here, we test predictions made by a recent computational model that accounts for a large set of studies examining goal-/sign-tracking behavior and the role that dopamine plays in learning. We show that increasing the duration of the time between trials leads more to the development of a sign-tracking response and to the release of dopamine in the nucleus accumbens. During conditioning with shorter intertrial intervals, goal tracking was more prominent, and dopamine was released upon presentation of the conditioned stimulus but not during the time of reward delivery after training. Thus, shorter intertrial intervals restored the classical dopamine reward prediction error pattern. Our results validate the computational hypothesis and open the door for understanding individual differences to classical conditioning.

Highly distributed neural circuits are thought to support adaptive decision-making in volatile and complex environments. Notably, the functional interactions between prefrontal and reciprocally connected thalamic nuclei areas may be important when choices are guided by current goal value or action-outcome contingency. We examined the functional involvement of selected thalamocortical and corticothalamic pathways connecting the dorsomedial prefrontal cortex (dmPFC) and the mediodorsal thalamus (MD) in the behaving rat. Using a chemogenetic approach to inhibit projection-defined dmPFC and MD neurons during an instrumental learning task, we show that thalamocortical and corticothalamic pathways differentially support goal attributes. Both pathways participate in adaptation to the current goal value, but only thalamocortical neurons are required to integrate current causal relationships. These data indicate that antiparallel flow of information within thalamocortical circuits may convey qualitatively distinct aspects of adaptive decision-making and highlight the importance of the direction of information flow within neural circuits.

The relationship between personality and learning abilities has become a growing field of interest. Studies have mainly focused on the relationship with performance, such as the speed of acquisition. In this study, we hypothesised that personality could in part also be related to a certain predisposition of an individual to switch more easily from a goal-directed process to a habit process during learning. To identify these processes, we conducted a contingency degradation protocol. This study investigated 1/ whether in general horses are able to adjust their response according to the contingency between their action and the reward, 2/ whether there are any relationships between certain personality profiles and a predisposition to switch more rapidly to habitual processes, and 3/ whether emotional states experienced during the learning procedure play a role in this switching. Personality tests were conducted on 29 horses, followed by a degradation contingency protocol. Overall, results show that horses were sensitive to contingency degradation between their action and the reward. Nevertheless, there was inter-individual variability: the horses presenting high fearfulness, and to a lesser extent low sensory sensitivity and low gregariousness were less sensitive to the degradation, demonstrating that they were more likely to switch to a habitual process. Contrary to our expectations, the emotional state experienced during the procedure did not seem to explain this switching. We conclude that personality is not only related to learning performance, but also in part to the process involved during learning, independently of the emotion experienced during the process. This study provides new theoretical knowledge on cognitive skills in ungulates.

There is a growing interest in determining the functional contribution of thalamic inputs to cortical functions. In the context of adaptive behaviours, identifying the precise role of the mediodorsal thalamus (MD) in particular remains difficult despite the large amount of experimental data available. A better understanding of the thalamocortical connectivity of this region may help to capture its functional role. To address this issue, this study focused exclusively on the specific connections from the MD to the prefrontal cortex (PFC) by means of direct comparisons of labelling produced by single and dual injections of retrograde tracers in the different subdivisions of the PFC in the rat. We show that at least three parallel and essentially separate thalamocortical pathways originate from the MD, as follows: projections to the dorsal (1) and the ventral (2) subdivisions of the mPFC follow a mediolateral topography at the thalamic level (i.e. medial thalamic neurons target the mPFC ventrally whereas lateral thalamic neurons project dorsally), whereas a considerable innervation to the OFC (3) includes thalamic cells projecting to both the lateral and the ventral OFC subdivisions. These observations provide new insight on the functions of the MD and suggest a specific focus on each of these pathways for future functional studies.