Andrew M. Wikenheiser

The progressive ratio (PR) schedule is a popular test of motivation. Despite its popularity, the PR task hinges on a low-dimensional behavioral readout—breakpoint or the maximum work requirement subjects are willing to complete before abandoning the task. Here, we show that with a simple modification, the PR task can be transformed into an optimization problem reminiscent of the patch-leaving foraging scenario, which has been analyzed extensively by behavioral ecologists, psychologists, and neuroscientists. In the PR with reset (PRR) task, male and female rats performed the PR task on one lever but could press a second lever to reset the current ratio requirement back to its lowest value at the cost of enduring a reset delay, during which both levers were retracted. Rats used the reset lever adaptively on the PRR task, and their ratio reset decisions were sensitive to the cost of the reset delay. We derived an approach for computing the optimal bout length—the number of rewards to earn before pressing the reset lever that produces the greatest long-term rate of reward—and found that rats flexibly changed their behavior to approximate the optimal strategy. However, rats showed a systematic bias for bout lengths that exceeded the optimal length, an effect reminiscent of “overharvesting” in patch-leaving tasks. The PRR task thus represents a novel means of testing how rats adapt to the cost–benefit structure of the environment in a way that connects deeply to the broader literature on associative learning and optimal foraging theory.

Animals in the wild must balance multiple, potentially mutually exclusive goals simultaneously in order to survive. Yet laboratory tests of decision-making often investigate how animals optimize their behavior to achieve a single, well-defined goal, which is often a nutritive reward. Thus, how animals solve multiobjective optimization problems is not well understood. Here, we devised an ethologically inspired decision-making task to examine how rats balance the pursuit of food and nonfood reinforcement. Rats performed a free-choice patch-foraging task, in which they could earn food in one location (food patch) or interact with a rodent play structure in a different location (toy patch). The cost of switching between patches was manipulated by requiring rats to endure a long or short “travel time” penalty during which they were not able to access either patch. Rats devoted a considerable amount of their limited foraging time to patches of both types, showing a small but significant preference for food patches. In accordance with theoretical models of foraging, when the cost of switching patches was high rats chose longer stay durations in both types of patches, suggesting that similar rational principles guided their pursuit of food and nonfood rewards. Examining the within-session dynamics of time allocation revealed that rats showed an early preference for spending time in toy patches that reversed over the course of the session. Satiety manipulations demonstrated that patch residence time decisions were under goal-directed control and responsive to current needs and recent consumption.

The Progressive Ratio (PR) schedule is a popular test for measuring the motivational value of a reinforcer, in which subjects must exert an increasing amount of work to obtain each successive reward. Despite its popularity, the PR task hinges on a low-dimensional behavioral readout—breakpoint, or the maximum work requirement subjects are willing to complete before abandoning the task. Here, we show that with a simple modification, the PR task can be transformed into an optimization problem reminiscent of the patch-leaving foraging scenario, which has been analyzed extensively by behavioral ecologists, psychologists, and neuroscientists. In the Progressive Ratio with Reset (PRR) task, rats perform the PR task on one lever, but can press a second lever to reset the current ratio requirement back to its lowest value at the cost of enduring a reset delay, during which both levers are retracted. Rats used the reset lever adaptively on the PRR task, and their ratio reset decisions were sensitive to the cost of the reset delay. We derived an approach for computing the optimal bout length—the number of rewards to earn before pressing the reset lever that produces the greatest long-term rate of reward—and found that rats flexibly changed their behavior to approximate the optimal strategy. However, rats showed a systematic bias for bout lengths that exceeded the optimal length, an effect reminiscent of “overharvesting” in patch-leaving tasks. The PRR task thus represents a novel means of testing whether and how rats adapt their behavior to the cost-benefit structure of the environment in a way that connects deeply to the broader literature on associative learning and optimal foraging theory.

Risk, or variance over outcomes, features prominently in many decisions, but the factors that determine when and how risk modulates decision making remain unclear. We tested how risk affected rats’ strategies for exploiting a diminishing food source as the overall richness of the environment was manipulated. Long-Evans rats earned food by sequentially visiting two foraging patches with different reward schedules—a low-variance standard option and a high-variance risky option—that provided the same average rate of reward. When rats switched between options, they encountered either a long or short delay during which no food was available to simulate the cost of travelling between patches. When the travel delay was short rats allocated more time to the low-variance standard reward schedule than the high-variance risky option. When the travel time was long rats spent the same amount of time in risky and standard patches. Consistent with previous work, rats “overharvested” patches, remaining for longer than the optimal patch residence duration. Overharvesting was prevalent in both risky and standard patches, and the magnitude of overharvesting increased with successive visits to the same patch, suggesting that overharvesting was not driven by uncertainty about the reward statistics of patches.

SUMMARY: Goal-directed decision making relies on learned action-outcome associations to allow actions to adapt when outcomes change. With repetition, behavioral control shifts toward habit, enabling more automatic action execution at the expense of flexibility. Much is unknown about how the brain learns and supports goal-directed behavior and what changes as habits form. Here, we longitudinally tracked and manipulated the activity of two major dorsomedial striatum populations, D1 ⁺ and D2/A2A ⁺ neurons, to reveal their activity and function during goal-directed learning and habit formation. Across learning, DMS D1 ⁺ neurons stably encode instrumental actions, carry information about behavioral engagement and organization, and develop and maintain action-reward representational similarity and conjunctive activation. This persists even after behavioral control transitions to habit. In contrast, one ensemble of DMS A2A ⁺ neurons transiently encodes actions during early learning and another slowly starts to encode actions as habits form. Both DMS D1 ⁺ and A2A ⁺ neuronal activity mediates the learning processes underlying goal-directed control, but only DMS D1 ⁺ neurons maintain support of adaptive, goal-directed decision making. These data reveal a cell-type-specific dissociation between stable v. reorganizing striatal action representations and, independently, persistent v. transient contributions to goal-directed control.

Electrophysiology has proven invaluable to record neural activity, and the development of Neuropixels probes dramatically increased the number of recorded neurons. These probes are often implanted acutely, but acute recordings cannot be performed in freely moving animals and the recorded neurons cannot be tracked across days. To study key behaviors such as navigation, learning, and memory formation, the probes must be implanted chronically. An ideal chronic implant should (1) allow stable recordings of neurons for weeks; (2) allow reuse of the probes after explantation; (3) be light enough for use in mice. Here, we present the “Apollo Implant”, an open-source and editable device that meets these criteria and accommodates up to two Neuropixels 1.0 or 2.0 probes. The implant comprises a “payload” module which is attached to the probe and is recoverable, and a “docking” module which is cemented to the skull. The design is adjustable, making it easy to change the distance between probes, the angle of insertion, and the depth of insertion. We tested the implant across eight labs in head-fixed mice, freely moving mice, and freely moving rats. The number of neurons recorded across days was stable, even after repeated implantations of the same probe. The Apollo implant provides an inexpensive, lightweight, and flexible solution for reusable chronic Neuropixels recordings.

Animals in the wild must balance multiple, potentially mutually-exclusive goals simultaneously in order to survive. Yet laboratory tests of decision making often investigate how animals optimize their behavior to achieve a single, well-defined goal, which is often a nutritive reward. Thus, how animals solve multi-objective optimization problems is not well understood. Here, we devised an ethologically-inspired decision making task to examine how rats balance the pursuit of food and non-food reinforcement. Rats performed a free-choice patch-foraging task, in which they could earn food in one location (food patch) or interact with a rodent play structure in a different location (toy patch). The cost of switching between patches was manipulated by requiring rats to endure a long or short “travel time” penalty during which they were not able to access either patch. Rats devoted a considerable amount of their limited foraging time to patches of both types, showing a small but significant preference for food patches. In accordance with theoretical models of foraging, when the cost of switching patches was high rats chose longer stay durations in both types of patches, suggesting that similar rational principles guided their pursuit of food and non-food rewards. Examining the within-session dynamics of time allocation revealed that rats showed an early preference for spending time in toy patches that reversed over the course of the session. Satiety manipulations demonstrated that patch residence time decisions were under goal-directed control, and responsive to current needs and recent consumption. These results validate a naturalistic approach to testing decision making in rats over a range of food and non-food goods. Key words: Patch foraging, decision making, multi-objective optimization, play, leisure Significance Statement: The mechanisms by which animals trade off competing goals remain poorly understood. Here, we investigated whether rats use rational decision-making strategies to balance essential needs with non-essential pursuits like leisure by letting rats choose between earning food and engaging with a play structure. Rats performing a free-choice patch-foraging task adjusted their behavior based on opportunity costs, staying longer in patches when switching costs were high, in accordance with models of optimal foraging. Rats allocated significant time to toy patches, even at the expense of food rewards, underscoring the intrinsic value of non-food reinforcers. These results validate a naturalistic approach for studying tradeoffs between qualitatively-different rewards, and advance our understanding of multi-goal optimization in the context of a naturalistic decision problem.

Orbitofrontal cortex (OFC) and secondary motor cortex (M2) are both implicated in flexible reward learning but the conditions that differentially recruit these regions are not fully understood. We imaged calcium activity from single neurons in OFC or M2 during de novo learning of increasingly uncertain reward probability schedules. Predictions of choice were decoded from M2 neurons with high accuracy under all certainty conditions, but were more accurately decoded from OFC neurons under greater uncertainty. The number of choice- and reward-selective neurons was significantly higher in M2 than in OFC across schedules, but these proportions increased across levels of uncertainty only in OFC. Decoding accuracy of choice and outcome was predicted by behavioral strategies Win-Stay and Lose-Shift in OFC, but not M2. We also tested causal involvement of these regions with chemogenetic perturbation as rats learned increasing and then decreasing uncertainty schedules. Whereas inhibition of OFC neurons attenuated learning across all schedules, M2 neurons were found to support learning in the most certain reward schedule. Thus, OFC neurons preferentially encode choices and outcomes that foster a greater reliance on adaptive strategies under uncertainty. This reveals a novel functional heterogeneity within frontal cortex in support of flexible learning.

Maladaptive decision-making is a hallmark of substance use disorders, though how drugs of abuse alter neural representations supporting adaptive behavior remains poorly understood. Past studies show the orbitofrontal (OFC) and prelimbic (PL) cortices are important for decision making, tracking both task-relevant and latent information. However, previous studies have focused on how drugs of abuse impact the firing rates of individual units. More work at the ensemble level is necessary to accurately characterize potential drug-induced changes. Using single-unit recordings in rats during a multidimensional decision-making task and then applying population and ensemble level analyses, we show that prior use of cocaine altered the strength and structure of task-relevant and latent representations in the OFC, changes relatable to suboptimal decision making in this and perhaps other settings. These data expand our understanding of the neuropathological underpinnings of maladaptive decision-making in SUDs, potentially enabling enhanced future treatment strategies.

The ubiquity, importance, and sophistication of foraging behavior makes it an ideal platform for studying naturalistic decision making in animals. We developed a spatial patch-foraging task for rats, in which subjects chose how long to remain in one foraging patch as the rate of food earnings steadily decreased. The cost of seeking out a new location was varied across sessions. The behavioral task was designed to mimic the structure of natural foraging problems, where distinct spatial locations are associated with different reward statistics, and decisions require navigation and movement through space. Male and female Long-Evans rats generally followed the predictions of theoretical models of foraging, albeit with a consistent tendency to persist with patches for too long compared to behavioral strategies that maximize food intake rate. The tendency to choose overly-long patch residence times was stronger in male rats. We also observed sex differences in locomotion as rats performed the task, but these differences in movement only partially accounted for the differences in patch residence durations observed between male and female rats. Together, these results suggest a nuanced relationship between movement, sex, and foraging decisions. Significance statement: Foraging behavior offers a naturalistic way of studying temporal investment amongst different choice options, a translationally-important form of decision making. Previous laboratory investigations have relied on foraging tasks that require little movement from subjects, which could affect the strategies that animals use and the neural mechanisms that support them. We developed a spatial foraging task for rats. Behavior generally matched the predictions of theoretical models, although rats remained in patches for longer than prescribed. Male rats exhibited a stronger tendency to overharvest patches than female rats. Sex differences in movement did not account for sex differences in foraging. These data highlight the interplay between movement and decision making, and demonstrate the utility of spatial tasks for studies of foraging.