Joseph J. Paton

Midbrain dopamine neurons (DANs) signal reward-prediction errors that teach recipient circuits about expected rewards¹. However, DANs are thought to provide a substrate for temporal difference (TD) reinforcement learning (RL), an algorithm that learns the mean of temporally discounted expected future rewards, discarding useful information about experienced distributions of reward amounts and delays². Here we present time–magnitude RL (TMRL), a multidimensional variant of distributional RL that learns the joint distribution of future rewards over time and magnitude. We also uncover signatures of TMRL-like computations in the activity of optogenetically identified DANs in mice during behaviour. Specifically, we show that there is significant diversity in both temporal discounting and tuning for the reward magnitude across DANs. These features allow the computation of a two-dimensional, probabilistic map of future rewards from just 450 ms of the DAN population response to a reward-predictive cue. Furthermore, reward-time predictions derived from this code correlate with anticipatory behaviour, suggesting that similar information is used to guide decisions about when to act. Finally, by simulating behaviour in a foraging environment, we highlight the benefits of a joint probability distribution of reward over time and magnitude in the face of dynamic reward landscapes and internal states. These findings show that rich probabilistic reward information is learnt and communicated to DANs, and suggest a simple, local-in-time extension of TD algorithms that explains how such information might be acquired and computed.

Because organisms are able to sense its passage, it is perhaps tempting to treat time as a sensory modality, akin to vision or audition. Indeed, certain features of sensory estimation, such as Weber's law, apply to timing and sensation alike (Gibbon, 1977; Pardo-Vazquez et al., 2019). However, from an organismal perspective, time is a derived feature of other signals, not a stimulus that can be readily transduced by sensory receptors. Its importance for biology lies in the fact that the physical world comprises a complex dynamical system. The multiscale spatiotemporal structure of sensory and internally generated signals within an organism is the informational fabric underlying its ability to control behavior. Viewed this way, temporal computations assume a more fundamental role than is implied by treating time as just another element of the experienced world (Paton & Buonomano, 2018). Thus, in this review we focus on temporal processing as a means of approaching the more general problem of how the nervous system produces adaptive behavior.

To behave adaptively, the brain must register temporal structure in the environment and use it to organize behavior. The dorsolateral striatum (DLS) integrates sensorimotor input, and is necessary for accurate timing and structuring behavior in general. However, if DLS provides the basis for mapping temporal features in the environment to behavior, its activity should predict variation in that mapping. A reanalysis of DLS population activity in rats comparing the duration of two sequentially presented vibratory stimuli revealed a striking correspondence between neural activity and behavior. Varying vibration intensity of the second stimulus induced systematic biases in temporal judgments, and corresponding biases in multiple features of DLS activity during stimulus presentation, including population coding of time. In contrast, the same intensity manipulations applied to the first stimulus affected neither behavior nor neural activity. Furthermore, neuronal response profiles were best described as a continuum, arguing against hypotheses where categories of responses, e.g., ramping activity, selectively underpin temporal processing. These data represent important additional evidence that striatal population dynamics support the organization of behavior by mapping temporal information to action.

Learning to predict rewards is a fundamental driver of adaptive behavior. Midbrain dopamine neurons (DANs) play a key role in such learning by signaling reward prediction errors (RPEs) that teach recipient circuits about expected rewards given current circumstances and actions. However, the algorithm that DANs are thought to provide a substrate for, temporal difference (TD) reinforcement learning (RL), learns the mean of temporally discounted expected future rewards, discarding useful information concerning experienced distributions of reward amounts and delays. Here we present time-magnitude RL (TMRL), a multidimensional variant of distributional reinforcement learning that learns the joint distribution of future rewards over time and magnitude using an efficient code that adapts to environmental statistics. In addition, we discovered signatures of TMRL-like computations in the activity of optogenetically identified DANs in mice during a classical conditioning task. Specifically, we found significant diversity in both temporal discounting and tuning for the magnitude of rewards across DANs, features that allow the computation of a two dimensional, probabilistic map of future rewards from just 450ms of neural activity recorded from a population of DANs in response to a reward-predictive cue. In addition, reward time predictions derived from this population code correlated with the timing of anticipatory behavior, suggesting the information is used to guide decisions regarding when to act. Finally, by simulating behavior in a foraging environment, we highlight benefits of access to a joint probability distribution of reward over time and magnitude in the face of dynamic reward landscapes and internal physiological need states. These findings demonstrate surprisingly rich probabilistic reward information that is learned and communicated to DANs, and suggest a simple, local-in-time extension of TD learning algorithms that explains how such information may be acquired and computed.

The basal ganglia are thought to contribute to decision-making and motor control. These functions are critically dependent on timing information, which can be extracted from the evolving state of neural populations in their main input structure, the striatum. However, it is debated whether striatal activity underlies latent, dynamic decision processes or kinematics of overt movement. Here, we measured the impact of temperature on striatal population activity and the behavior of rats, and compared the observed effects with neural activity and behavior collected in multiple versions of a temporal categorization task. Cooling caused dilation, and warming contraction, of both neural activity and patterns of judgment in time, mimicking endogenous decision-related variability in striatal activity. However, temperature did not similarly affect movement kinematics. These data provide compelling evidence that the timecourse of evolving striatal activity dictates the speed of a latent process that is used to guide choices, but not continuous motor control. More broadly, they establish temporal scaling of population activity as a likely neural basis for variability in timing behavior.

The direct and indirect pathways of the basal ganglia are classically thought to promote and suppress action, respectively¹. However, the observed co-activation of striatal direct and indirect medium spiny neurons² (dMSNs and iMSNs, respectively) has challenged this view. Here we study these circuits in mice performing an interval categorization task that requires a series of self-initiated and cued actions and, critically, a sustained period of dynamic action suppression. Although movement produced the co-activation of iMSNs and dMSNs in the sensorimotor, dorsolateral striatum (DLS), fibre photometry and photo-identified electrophysiological recordings revealed signatures of functional opponency between the two pathways during action suppression. Notably, optogenetic inhibition showed that DLS circuits were largely engaged to suppress—and not promote—action. Specifically, iMSNs on a given hemisphere were dynamically engaged to suppress tempting contralateral action. To understand how such regionally specific circuit function arose, we constructed a computational reinforcement learning model that reproduced key features of behaviour, neural activity and optogenetic inhibition. The model predicted that parallel striatal circuits outside the DLS learned the action-promoting functions, generating the temptation to act. Consistent with this, optogenetic inhibition experiments revealed that dMSNs in the associative, dorsomedial striatum, in contrast to those in the DLS, promote contralateral actions. These data highlight how opponent interactions between multiple circuit- and region-specific basal ganglia processes can lead to behavioural control, and establish a critical role for the sensorimotor indirect pathway in the proactive suppression of tempting actions.

Reward expectations based on internal knowledge of the external environment are a core component of adaptive behavior. However, internal knowledge may be inaccurate or incomplete due to errors in sensory measurements. Some features of the environment may also be encoded inaccurately to minimize representational costs associated with their processing. In this study, we investigated how reward expectations are affected by features of internal representations by studying behavior and dopaminergic activity while mice make time-based decisions. We show that several possible representations allow a reinforcement learning agent to model animals’ overall performance during the task. However, only a small subset of highly compressed representations simultaneously reproduced the co-variability in animals’ choice behavior and dopaminergic activity. Strikingly, these representations predict an unusual distribution of response times that closely match animals’ behavior. These results inform how constraints of representational efficiency may be expressed in encoding representations of dynamic cognitive variables used for reward-based computations.

The basal ganglia (BG) are thought to contribute to decision-making and motor control by influencing action selection based on consequences. These functions are critically dependent on timing information that can be extracted from the evolving state of neural populations in the striatum, the major input area of the BG. However, it is debated whether striatal activity underlies latent, dynamic decision processes or kinematics of overt movement. Here, we measured the impact of temperature on striatal population activity and the behavior of rats and compared the observed effects to neural activity and behavior collected in multiple versions of a temporal categorization task. Cooler temperatures caused dilation, and warmer temperatures contraction, of both neural activity and patterns of judgment in time, mimicking endogenous decision-related variability in striatal activity. However, temperature did not similarly affect movement kinematics. These data provide compelling evidence that the time course of evolving striatal population activity dictates the speed of a latent process that is used to guide choices, but not moment by moment kinematics. More broadly, they establish temporal scaling of population activity as a likely cause and not simply a correlate of timing behavior in the brain.