RLRichard L. Lewis

Richard L. Lewis

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Richard L. Lewis has a total of 18 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Do Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm

    by Xiaoyang Hu, Richard L. Lewis

    Xiaoyang Hu, Richard Lewis. Findings of the Association for Computational Linguistics: ACL 2025. 2025.


  • Open Access

    Adaptive Pairwise Weights for Temporal Credit Assignment

    by Zeyu Zheng, Risto Vuorio, Richard Lewis, Satinder Singh

    How much credit (or blame) should an action taken in a state get for a future reward? This is the fundamental temporal credit assignment problem in Reinforcement Learning (RL). One of the earliest and still most widely used heuristics is to assign this credit based on a scalar coefficient, $\lambda$ (treated as a hyperparameter), raised to the power of the time interval between the state-action and the reward. In this empirical paper, we explore heuristics based on more general pairwise weightings that are functions of the state in which the action was taken, the state at the time of the reward, as well as the time interval between the two. Of course it isn't clear what these pairwise weight functions should be, and because they are too complex to be treated as hyperparameters we develop a metagradient procedure for learning these weight functions during the usual RL training of a policy. Our empirical work shows that it is often possible to learn these pairwise weight functions during learning of the policy to achieve better performance than competing approaches.

  • Open Access

    Learning State Representations from Random Deep Action-conditional Predictions

    by Zeyu Zheng, Vivek Veeriah, Risto Vuorio, Richard Lewis, Satinder Singh

    Our main contribution in this work is an empirical finding that random General Value Functions (GVFs), i.e., deep action-conditional predictions -- random both in what feature of observations they predict as well as in the sequence of actions the predictions are conditioned upon -- form good auxiliary tasks for reinforcement learning (RL) problems. In particular, we show that random deep action-conditional predictions when used as auxiliary tasks yield state representations that produce control performance competitive with state-of-the-art hand-crafted auxiliary tasks like value prediction, pixel control, and CURL in both Atari and DeepMind Lab tasks. In another set of experiments we stop the gradients from the RL part of the network to the state representation learning part of the network and show, perhaps surprisingly, that the auxiliary tasks alone are sufficient to learn state representations good enough to outperform an end-to-end trained actor-critic baseline. We opensourced our code at https://github.com/Hwhitetooth/random_gvfs.

  • Rational use of episodic and working memory: A normative account of prospective memory

    by Ida Momennejad, Jarrod Lewis-Peacock, Kenneth A. Norman, Jonathan D. Cohen, Satinder Singh, Richard L. Lewis


  • Cognitively bounded rational analyses and the crucial role of theories of subjective utility

    by Richard L. Lewis, Andrew Howes

    We agree that combining rational analysis with cognitive bounds, what we previously introduced as Cognitively Bounded Rational Analysis, is a promising and under-used methodology in psychology. We further situate the framework in the literature, and highlight the important issue of a theory of subjective utility, which is not addressed sufficiently clearly in the framework or related previous work.

  • Open Access

    How Should an Agent Practice?

    by Janarthanan Rajendran, Richard Lewis, Vivek Veeriah, Honglak Lee, Satinder Singh

    We present a method for learning intrinsic reward functions to drive the learning of an agent during periods of practice in which extrinsic task rewards are not available. During practice, the environment may differ from the one available for training and evaluation with extrinsic rewards. We refer to this setup of alternating periods of practice and objective evaluation as practice-match, drawing an analogy to regimes of skill acquisition common for humans in sports and games. The agent must effectively use periods in the practice environment so that performance improves during matches. In the proposed method the intrinsic practice reward is learned through a meta-gradient approach that adapts the practice reward parameters to reduce the extrinsic match reward loss computed from matches. We illustrate the method on a simple grid world, and evaluate it in two games in which the practice environment differs from match: Pong with practice against a wall without an opponent, and PacMan with practice in a maze without ghosts. The results show gains from learning in practice in addition to match periods over learning in matches only.

  • Open Access

    Rational use of Episodic and Working Memory: A Normative Account of Prospective Memory

    by Ida Momennejad, Jarrod Lewis-Peacock, Kennneth A. Norman, Jonathan D Cohen, Satinder Singh, Richard L. Lewis

    A bstract: Humans often simultaneously pursue multiple plans at different time scales. The successful realization of non-immediate plans (e.g., post package after work) requires keeping track of a future plan while accomplishing other intermediate tasks (e.g., write a paper), a capacity known as prospective memory . This capacity requires the integration of noisy evidence from perceptual input with evidence from short-term working memory (WM) and longer-term or episodic memory (LTM/EM). Here we formulate a set of dual-task problems in empirical studies of prospective memory as problems of computational rationality, and ask how a rational model should exploit noisy perception and memory to maximize payoffs. The model combines reinforcement learning (optimal action selection) with evidence accumulation (optimal inference) in order to derive good decision parameters for optimal task performance (i.e., performing an ongoing task while monitoring for a cue that triggers executing a second prospective task). We compare model behavior to key accuracy and reaction time phenomena in human performance. Thus, we offer a normative approach to theorizing and modeling these phenomena without assumptions about mechanisms of attention or retrieval. This approach can be extended to study meta-parameters governing the boundedly rational use of memory in planned action in health, as well as compensatory mnemonic strategies that may be rational responses to disturbances of these mechanisms in neuropsychiatric disorders.

  • The role of (bounded) optimization in theory testing and prediction

    by Andrew Howes, Richard L. Lewis

    We argue that a radically increased emphasis on (bounded) optimality can contribute to cognitive science by supporting prediction. Bounded optimality (computational rationality), an idea that borrowed from artificial intelligence, supports a priori behavioral prediction from constrained generative models of cognition. Bounded optimality thereby addresses serious failings with the logic and testing of descriptive models of perception and action.

  • Top Cited Papers

    Computational Rationality: Linking Mechanism and Behavior Through Bounded Utility Maximization

    by Richard L. Lewis, Andrew Howes, Satinder Singh

    We propose a framework for including information‐processing bounds in rational analyses. It is an application of bounded optimality (Russell & Subramanian, 1995) to the challenges of developing theories of mechanism and behavior. The framework is based on the idea that behaviors are generated by cognitive mechanisms that are adapted to the structure of not only the environment but also the mind and brain itself. We call the framework computational rationality to emphasize the incorporation of computational mechanism into the definition of rational action. Theories are specified as optimal program problems, defined by an adaptation environment, a bounded machine, and a utility function. Such theories yield different classes of explanation, depending on the extent to which they emphasize adaptation to bounds, and adaptation to some ecology that differs from the immediate local environment. We illustrate this variation with examples from three domains: visual attention in a linguistic task, manual response ordering, and reasoning. We explore the relation of this framework to existing “levels” approaches to explanation, and to other optimality‐based modeling approaches.


  • Top Cited Papers

    Computational principles of working memory in sentence comprehension

    by Richard L. Lewis, Shravan Vasishth, Julie A. Van Dyke