Matthew Tong

Although a standard reinforcement learning model can capture many aspects of reward-seeking behaviors, it may not be practical for modeling human natural behaviors because of the richness of dynamic environments and limitations in cognitive resources. We propose a modular reinforcement learning model that addresses these factors. Based on this model, a modular inverse reinforcement learning algorithm is developed to estimate both the rewards and discount factors from human behavioral data, which allows predictions of human navigation behaviors in virtual reality with high accuracy across different subjects and with different tasks. Complex human navigation trajectories in novel environments can be reproduced by an artificial agent that is based on the modular model. This model provides a strategy for estimating the subjective value of actions and how they influence sensory-motor decisions in natural behavior. It is generally agreed that human actions can be formalized within the framework of statistical decision theory, which specifies a cost function for actions choices, and that the intrinsic value of actions is controlled by the brain’s dopaminergic reward machinery. Given behavioral data, the underlying subjective reward value for an action can be estimated through a machine learning technique called inverse reinforcement learning. Hence it is an attractive method for studying human reward-seeking behaviors. Standard reinforcement learning methods were developed for artificial intelligence agents, and incur too much computation to be a viable model for real-time human decision making. We propose an approach called modular reinforcement learning that decomposes a complex task into independent decision modules. This model includes a frequently overlooked variable called the discount factor, which controls the degree of impulsiveness in seeking future reward. We develop an algorithm called modular inverse reinforcement learning that estimates both the reward and the discount factor. We show that modular reinforcement learning may be a useful model for natural navigation behaviors. The estimated rewards and discount factors explain human walking direction decisions in a virtual-reality environment, and can be used to train an artificial agent that can accurately reproduce human navigation trajectories.

Search is a central visual function. Most of what is known about search derives from experiments where subjects view 2D displays on computer monitors. In the natural world, however, search involves movement of the body in large-scale spatial contexts, and it is unclear how this might affect search strategies. In this experiment, we explore the nature of memory representations developed when searching in an immersive virtual environment. By manipulating target location, we demonstrate that search depends on episodic spatial memory as well as learnt spatial priors. Subjects rapidly learned the large-scale structure of the space, with shorter paths and less head rotation to find targets. These results suggest that spatial memory of the global structure allows a search strategy that involves efficient attention allocation based on the relevance of scene regions. Thus spatial memory may allow less energetically costly search strategies.

In everyday search, both visual information and spatial memory are used, even when targets are within the field of view. Yet little is known about the relative importance of these two kinds of cue and how they work together. It seems likely that spatial memory representations guide not only eye movements but also head and body movements (Aivar et al., 2015; Won et al., 2015). We explored this issue in a large-scale immersive environment. Subjects searched for targets located on four surfaces in each of the two rooms in a virtual reality apartment. At the start of each trial, the target was presented on a TV at the end of the hallway separating two rooms. Subjects turned around to make either a left or right turn to enter a room. To test if memory aided search by guiding head movements once they entered the room, we analyzed the angles between head and target direction while making the first fixation to the target. The results showed that the angles from the head to target became about 24 degrees smaller after only three repetitions of search (at least for the target located at locations that are easiest to orient to upon entrance). The next question is whether memory could help prepare the body to move in the right direction even before entering the room. The angles between head and target direction one second before entering the room were analyzed. Smaller angles were found with experience (about 11 degrees), even though targets were not visible. Together, the findings suggested that spatial memory for the target location is used for advanced planning of body movements, even before the search scene is visible. Thus making movement decisions based on memory allows for more efficient search when body movements are involved. Meeting presented at VSS 2017

In the context of natural behaviors, humans must gather information to choose between different task needs, such as avoiding obstacles or heading towards a target. The brain's neural reward machinery has been implicated in these action choices, and a technique called Inverse Reinforcement Learning (IRL) can be used to estimate reward functions from behavioral data. A frequently overlooked variable in IRL is the discount factor: how much a future reward matters compared to the current reward. If future rewards are too heavily discounted, a person could overlook a future obstacle, even though it incurs a large negative reward. We argue that reward, together with discount factor, defines a value surface for a single task. The reward controls the maximum height of the surface, and the discount factor controls how fast the reward decreases over time or distance, i.e., surface shape. Value surfaces are computationally easy to compose, hence multi-task behaviors can be modeled by combining these surfaces. This leads naturally to a divide-and-conquer approach of IRL, called modular IRL, which estimates relative rewards for subtasks. We expand upon previous modular IRL models (Rothkopf and Ballard, 2013) to include estimating the discount factor, and justify its correctness theoretically and experimentally through computer simulations as well as human experiments. We collect human navigation data in a virtual reality environment. Subjects are instructed to do a combination of following a path, collecting targets, and avoiding obstacles. We show the rewards and discount factors estimated from our algorithm reflect task instructions, and can accurately predict human actions (average angular difference = 24°). Furthermore, with two variables per objective (reward and discount factor), a virtual agent is able to reproduce long human-like navigation trajectories through the environment. We conclude that modular IRL with learned discount factors could be a powerful model for multi-task sensorimotor behaviors. Meeting abstract presented at VSS 2017

While it is universally acknowledged that both bottom up and top down factors contribute to allocation of gaze, we currently have limited understanding of how top-down factors determine gaze choices in the context of ongoing natural behavior. One purely top-down model by Sprague, Ballard, and Robinson (2007) suggests that natural behaviors can be understood in terms of simple component behaviors, or modules, that are executed according to their reward value, with gaze targets chosen in order to reduce uncertainty about the particular world state needed to execute those behaviors. We explore the plausibility of the central claims of this approach in the context of a task where subjects walk through a virtual environment performing interceptions, avoidance, and path following. Many aspects of both walking direction choices and gaze allocation are consistent with this approach. Subjects use gaze to reduce uncertainty for task-relevant information that is used to inform action choices. Notably the addition of motion to peripheral objects did not affect fixations when the objects were irrelevant to the task, suggesting that stimulus saliency was not a major factor in gaze allocation. The modular approach of independent component behaviors is consistent with the main aspects of performance, but there were a number of deviations suggesting that modules interact. Thus the model forms a useful, but incomplete, starting point for understanding top-down factors in active behavior.

Subjects lack perfect knowledge about the world, so to understand behavior, it is necessary to take subjects' uncertainty into account. Sprague et al (2007) suggested that gaze is directed to locations in order to resolve uncertainty affecting task performance, with uncertainty about unattended parts of the scene increasing over time. This uncertainty can be due to intrinsic factors (like motor noise and imperfect memory) and extrinsic factors (a dynamic world state). We evaluated the consequences of intrinsic and extrinsic uncertainty, revealed in the behavioral choices subjects made while walking through a cluttered scene. Subjects followed a path in a virtual environment while intercepting targets and avoiding obstacles. Extrinsic uncertainty was manipulated by adding random motion to targets and obstacles. We examined gaze and walking behavior of 12 subjects while they avoided obstacles. We plotted a subject's path from the time that fixation on an obstacle ended, and measured the minimal distance at which the subject subsequently passed the obstacle. When obstacles are stationary subjects give wider clearance when fixation had ended at a greater distance. Modal clearance was 0.42m for fixations ending with obstacles about 0.4m away, increasing to 0.55m for fixations that ended when the subject was further than 2.25m. This suggests that subjects take into account their intrinsic uncertainty, which accumulates over the greater distance walked after fixation ended. When the obstacles were moving, the distance of the last fixation had a greater effect (increasing to 0.64m clearance at large distance), suggesting that the extrinsic uncertainty about scene state is also taken into account. Together, this suggests that both kinds of uncertainty are taken into account when planning path trajectories (consistent with the decision-theoretic account described by Wolpert and Landy, 2012), and also that the position of the obstacle is only imperfectly updated when it is not fixated. Meeting presented at VSS 2016

The role of memory in guiding attention in real scenes is complex. One way that experience might guide search is by reducing the areas of the scene searched and directing search away from irrelevant regions (Neider and Zelinsky, 2008, Wolfe et al., 2011). We explored whether this is a significant factor in guiding search in immersive natural environments. We performed the search task in a 3D virtual environment, as well as in a 2D version that was parallel in task structure. Subjects searched for targets that were located on one of four surfaces in either of two rooms. We investigated the source of improvement from learning the space at both global and local levels: i) which room contains the target and ii) in which part of that room the target is located. We found that the probability of choosing the correct room to search on first entry is higher in 3D than in 2D, and improves in both cases. The number of room entries required to find the targets was also lower in 3D and decreased over episodes. More fixations inside the rooms are directed to surfaces that potentially contain targets in both 2D and 3D (increased from 60 to 87 of fixations after 24 trials in 3D). Surface fixations even early in the experiment indicate the influence of prior scene knowledge. Together these results indicate that memory representations facilitate search by restricting attention to the relevant parts of the environment while excluding the irrelevant parts at the global level (choosing the correct room) and also at the local level (looking at relevant regions inside the room). In addition, the data also suggest that global memory is used more in 3D, potentially because of the higher energetic or temporal cost of changing the region to be searched. Meeting abstract presented at VSS 2016

Previous research has shown that information that is task-relevant is retained over time and protected from posterior interference (e.g. Maxcey-Richard & Hollingworth, 2013). To examine whether location information is acquired and persists over the time course of natural actions we had participants perform a visual search task in an immersive virtual reality apartment. Participants searched the two rooms that formed the apartment for a series of geometric target objects, which were always visible. Eye movements, head movements and body trajectories were recorded. Target locations were manipulated to see if participants took advantage of regularities in the environment. Specifically, all target objects (8) were presented at the same locations in blocks 1, 2, 3 and 5, but switched location in each trial during block 4. We analyzed search time and number of errors (visits to wrong room) per block of 8 trials. Average search time decreased significantly between blocks 1 and 3 and then increased significantly in block 4. Interestingly, search time decreased again in block 5, once objects returned to their locations, with averages significantly lower than those of block 3. Room errors increased to 50% in block 4 but decreased significantly in block 5 to values similar to those of block 3. Gaze content also showed that about half of the participants used a memory-based strategy for finding the objects in block 4: on 28% of the trials these participants' first object fixation was on the object now occupying the location where the target used to be in blocks 1 to 3. From these three results we conclude that participants retained target location information from previous trials and used this information to make their search more efficient (i.e. block 5). The unreliability of object locations during block 4 was not enough to overwrite the information acquired during the first three blocks. Meeting presented at VSS 2016

The role of memory in guiding attention allocation in daily behaviors is not well understood. In experiments with two-dimensional (2D) images, there is mixed evidence about the importance of memory. Because the stimulus context in laboratory experiments and daily behaviors differs extensively, we investigated the role of memory in visual search, in both two-dimensional (2D) and three-dimensional (3D) environments. A 3D immersive virtual apartment composed of two rooms was created, and a parallel 2D visual search experiment composed of snapshots from the 3D environment was developed. Eye movements were tracked in both experiments. Repeated searches for geometric objects were performed to assess the role of spatial memory. Subsequently, subjects searched for realistic context objects to test for incidental learning. Our results show that subjects learned the room-target associations in 3D but less so in 2D. Gaze was increasingly restricted to relevant regions of the room with experience in both settings. Search for local contextual objects, however, was not facilitated by early experience. Incidental fixations to context objects do not necessarily benefit search performance. Together, these results demonstrate that memory for global aspects of the environment guides search by restricting allocation of attention to likely regions, whereas task relevance determines what is learned from the active search experience. Behaviors in 2D and 3D environments are comparable, although there is greater use of memory in 3D.

In everyday behavior there are multiple competing demands for gaze; for instance, walking along a sidewalk requires paying attention to the path while avoiding other pedestrians. Therefore, humans make numerous fixations to satisfy their behavioral goals. One attempt to explore suitable strategies of gaze allocation for natural situations was made by Sprague, Ballard and Robinson (2007), whose computational model predicts human visuo-motor behavior based on intrinsic reward and uncertainty. Their model presupposes that specific visual information is acquired from the currently fixated object only. However, evidence that peripheral objects affect gaze position by drawing it towards the center of gravity of target ensembles (Findlay, 1982; Vishwanath & Kowler, 2003) challenges this premise. Since the influence of peripheral information for natural vision remains largely unexplored, we investigated whether gaze targeting is biased towards peripheral objects in naturalistic tasks. Using a Virtual Reality environment, the fixations of 12 participants were examined while they walked through a virtual room with objects of two different colors that were designated targets or obstacles. The subjects were instructed to either collect targets, to avoid obstacles or to do both tasks simultaneously. For situations in which one of two visible objects was fixated, subjects’ gaze positions were biased more often towards the non-fixated object (72.6%) than away from it (27.4%). Moreover, the gaze position tended to be drawn more frequently towards the neighboring object when this neighbor was relevant to the current task than when it was task-irrelevant. These results indicate that information from peripheral objects affect human gaze targeting in natural vision. Furthermore, the effect of the neighbor’s task-relevance - and therefore of intrinsic reward - suggests that in a given fixation subjects might gather information from the peripheral visual field to accomplish a current set of goals at once. Meeting abstract presented at VSS 2015