François Cinotti

Modélisation computationnelle de la variabilité et de la régulation de l'apprentissage par renforcement et des signaux dopaminergiques associés chez le rat: Dans cette thèse, je traiterai de deux sujets principaux concernant les comportements d’apprentissage chez le Rat : premièrement, ce qu’il convient d’appeler le méta-apprentissage ("meta-learning" en anglais), c’est-à-dire la capacité d’autorégulation des paramètres comportementaux qui déterminent la prise de décision ;deuxièmement, la variabilité inter-individuelle quant au choix de la stratégie à employer dans le cadre d’une expérience de conditionnement pavlovien. Pour chacune de ces deux thématiques, j’adopterai un point de vue computationnel ancré dansles techniques de l’apprentissage par renforcement afin de modéliser des données expérimentales, tout en dressant des parallèles avec les fonctions dopaminergiques censées être associées à ces processus.

Common sense tells us that biological systems are goal-directed, and yet the concept remains philosophically problematic. We propose a novel characterization of goal-directed activities as a basis for hypothesizing about and investigating explanatory mechanisms. We focus on survival goals such as providing adequate nutrition to body tissues, highlighting two key features—normativity and action. These are closely linked inasmuch as goal-directed actions must meet normative requirements such as that they occur when required and not at other times. We illustrate how goal-directed actions are initiated and terminated not by environmental features and goals themselves, but by markers for them. For example, timely blood clotting is the essential response to injury, but platelet activation, required for clotting, is initiated not by the injury itself but by a short sequence of amino acids (GPO) that provides a reliable marker for it. We then make the case that the operation of markers is a prerequisite for common biological phenomena such as mistake-proneness and mimicry. We go on to identify properties of markers in general, including those that are genetically determined and those that can be acquired through associative learning. Both provide the basis for matching actions to changing environments and hence adaptive goal-directedness. We describe how goal-directed activities such as bird nest construction and birdsong learning, completed in anticipation of actions in the environment, have to be evaluated and practiced against a standard of correctness. This characterization of goal-directedness is sufficiently detailed to provide a basis for the scientific study of mechanisms.

The making of mistakes by organisms and other living systems is a theoretically and empirically unifying feature of biological investigation. Mistake theory is a rigorous and experimentally productive way of understanding this widespread phenomenon. It does, however, run up against the long-standing “functions” debate in philosophy of biology. Against the objection that mistakes are just a kind of malfunction, and that without a position on functions there can be no theory of mistakes, we reply that this is to misunderstand the theory. In this paper we set out the basic concepts of mistake theory and then argue that mistakes are a distinctive phenomenon in their own right, not just a kind of malfunction. Moreover, the functions debate is, to a large degree, independent of the concept of biological mistakes we outline. In particular, although the popular selected effects theory may retain its place within a more pluralistic conception of biological function, there is also need for a more forward-looking approach, where a robust concept of normativity can be an important driver of future experimental work.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade‐off has been extensively studied in pretrained animals facing non‐stationary decision‐making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long‐term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three‐armed bandit task across 24 days of pretraining. We found that the day‐by‐day evolution of rat performance and win‐shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta‐learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to exploit what they currently believe to be their best option, or instead explore potential alternatives in case better opportunities are in fact available. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while progressively learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilization of the way they regulated the exploration-exploitation trade-off. We successfully captured these behavioural adaptations using a meta-learning model in which the exploration-exploitation trade-off is controlled by the animal's average reward rate.

Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENTModel-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.

Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based, instrumental tasks parallel those engaged by Pavlovian based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, i.e., sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within- subject design to assess sign-tracking and goal-tracking behaviors using a Pavlovian conditioned approach task, and, then characterized behavior using an instrumental multi-stage decision-making (MSDM) task in rats. We hypothesized that both Pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that Pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states. Significance Statement: Model-free and model-based computations that guide instrumental, decision-making processes may also be recruited in Pavlovian based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both Pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign- tracking and goal-tracking behaviors were assessed in rats using a Pavlovian conditioned approach task, and, then instrumental behavior characterized using a multi- stage decision-making (MSDM) task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that Pavlovian and instrumental behaviors are driven by conserved reinforcement-learning mechanisms.

The striatum's complex microcircuit is made by connections within and between its D1- and D2-receptor expressing projection neurons and at least five species of interneuron. Precise knowledge of this circuit is likely essential to understanding striatum's functional roles and its dysfunction in a wide range of movement and cognitive disorders. We introduce here a Bayesian approach to mapping neuron connectivity using intracellular recording data, which lets us simultaneously evaluate the probability of connection between neuron types, the strength of evidence for it, and its dependence on distance. Using it to synthesize a complete map of the mouse striatum, we find strong evidence for two asymmetries: a selective asymmetry of projection neuron connections, with D2 neurons connecting twice as densely to other projection neurons than do D1 neurons, but neither subtype preferentially connecting to another; and a length-scale asymmetry, with interneuron connection probabilities remaining non-negligible at more than twice the distance of projection neuron connections. We further show that our Bayesian approach can evaluate evidence for wiring changes, using data from the developing striatum and a mouse model of Huntington's disease. By quantifying the uncertainty in our knowledge of the microcircuit, our approach reveals a wide range of potential striatal wiring diagrams consistent with current data.SIGNIFICANCE STATEMENTTo properly understand a neuronal circuit's function, it is important to have an accurate picture of the rate of connection between individual neurons and how this rate changes with the distance separating pairs of neurons. We present a Bayesian method for extracting this information from experimental data and apply it to the mouse striatum, a subcortical structure involved in learning and decision-making, which is made up of a variety of different projection neurons and interneurons. Our resulting statistical map reveals not just the most robust estimates of the probability of connection between neuron types, but also the strength of evidence for them, and their dependence on distance.

The striatum’s complex microcircuit is made by connections within and between its D1- and D2-receptor expressing projection neurons and at least five species of interneuron. Precise knowledge of this circuit is likely essential to understanding striatum’s functional roles and its dysfunction in a wide range of movement and cognitive disorders. We introduce here a Bayesian approach to mapping neuron connectivity using intracellular recording data, which lets us simultaneously evaluate the probability of connection between neuron types, the strength of evidence for it, and its dependence on distance. Using it to synthesise a complete map of the mouse striatum, we find strong evidence for two asymmetries: a selective asymmetry of projection neuron connections, with D2 neurons connecting twice as densely to other projection neurons than do D1 neurons, but neither subtype preferentially connecting to another; and a length-scale asymmetry, with interneuron connection probabilities remaining non-negligible at more than twice the distance of projection neuron connections. We further show our Bayesian approach can evaluate evidence for wiring changes, using data from the developing striatum and a mouse model of Huntington’s disease. By quantifying the uncertainty in our knowledge of the microcircuit, our approach reveals a wide range of potential striatal wiring diagrams consistent with current data.

In the context of Pavlovian conditioning, two types of behaviour may emerge within the population (Flagel et al. Nature, 469(7328): 53–57, 2011 ). Animals may choose to engage either with the conditioned stimulus (CS), a behaviour known as sign-tracking (ST) which is sensitive to dopamine inhibition for its acquisition, or with the food cup in which the reward or unconditioned stimulus (US) will eventually be delivered, a behaviour known as goal-tracking (GT) which is dependent on dopamine for its expression only. Previous work by Lesaint et al. (PLoS Comput Biol, 10(2), 2014 ) offered a computational explanation for these phenomena and led to the prediction that varying the duration of the inter-trial interval (ITI) would change the relative ST-GT proportion in the population as well as phasic dopamine responses. A recent study verified this prediction, but also found a rich variance of ST and GT behaviours within the trial which goes beyond the original computational model. In this paper, we provide a computational perspective on these novel results.