Richard L. Lewis
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Richard L. Lewis has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
- Satinder Singh
University of Michigan
5 shared publications - Andrew Howes
School of Computer Science, University of Birmingham, United Kingdom
2 shared publications - Jonathan D. Cohen
Princeton Neuroscience Institute, Princeton University, Princeton, USA
2 shared publications - Kenneth Andrew Norman
Princeton Neuroscience Institute, Princeton University, Princeton, USA
2 shared publications - Risto Vuorio
University of Oxford
2 shared publications - Shravan Vasishth
University of Potsdam
2 shared publications
Latest Publications
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 2665-2677
Do Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm
Xiaoyang Hu, Richard Lewis. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 2665-2677
Open AccessDo Language Models Understand the Cognitive Tasks Given to Them? Investigations with the N-Back Paradigm
Xiaoyang Hu, Richard Lewis. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- Open Access
Adaptive Pairwise Weights for Temporal Credit Assignment
How much credit (or blame) should an action taken in a state get for a future reward? This is the fundamental temporal credit assignment problem in Reinforcement Learning (RL). One of the earliest and still most widely used heuristics is to assign this credit based on a scalar coefficient, $\lambda$ (treated as a hyperparameter), raised to the power of the time interval between the state-action and the reward. In this empirical paper, we explore heuristics based on more general pairwise weightings that are functions of the state in which the action was taken, the state at the time of the reward, as well as the time interval between the two. Of course it isn't clear what these pairwise weight functions should be, and because they are too complex to be treated as hyperparameters we develop a metagradient procedure for learning these weight functions during the usual RL training of a policy. Our empirical work shows that it is often possible to learn these pairwise weight functions during learning of the policy to achieve better performance than competing approaches.Open AccessAdaptive Pairwise Weights for Temporal Credit Assignment
How much credit (or blame) should an action taken in a state get for a future reward? This is the fundamental temporal credit assignment problem in Reinforcement Learning (RL). One of the earliest and still most widely used heuristics is to assign this credit based on a scalar coefficient, $\lambda$ (treated as a hyperparameter), raised to the power of the time interval between the state-action and the reward. In this empirical paper, we explore heuristics based on more general pairwise weightings that are functions of the state in which the action was taken, the state at the time of the reward, as well as the time interval between the two. Of course it isn't clear what these pairwise weight functions should be, and because they are too complex to be treated as hyperparameters we develop a metagradient procedure for learning these weight functions during the usual RL training of a policy. Our empirical work shows that it is often possible to learn these pairwise weight functions during learning of the policy to achieve better performance than competing approaches. - Open Access
Learning State Representations from Random Deep Action-conditional Predictions
Our main contribution in this work is an empirical finding that random General Value Functions (GVFs), i.e., deep action-conditional predictions -- random both in what feature of observations they predict as well as in the sequence of actions the predictions are conditioned upon -- form good auxiliary tasks for reinforcement learning (RL) problems. In particular, we show that random deep action-conditional predictions when used as auxiliary tasks yield state representations that produce control performance competitive with state-of-the-art hand-crafted auxiliary tasks like value prediction, pixel control, and CURL in both Atari and DeepMind Lab tasks. In another set of experiments we stop the gradients from the RL part of the network to the state representation learning part of the network and show, perhaps surprisingly, that the auxiliary tasks alone are sufficient to learn state representations good enough to outperform an end-to-end trained actor-critic baseline. We opensourced our code at https://github.com/Hwhitetooth/random_gvfs.
Open AccessLearning State Representations from Random Deep Action-conditional Predictions
Our main contribution in this work is an empirical finding that random General Value Functions (GVFs), i.e., deep action-conditional predictions -- random both in what feature of observations they predict as well as in the sequence of actions the predictions are conditioned upon -- form good auxiliary tasks for reinforcement learning (RL) problems. In particular, we show that random deep action-conditional predictions when used as auxiliary tasks yield state representations that produce control performance competitive with state-of-the-art hand-crafted auxiliary tasks like value prediction, pixel control, and CURL in both Atari and DeepMind Lab tasks. In another set of experiments we stop the gradients from the RL part of the network to the state representation learning part of the network and show, perhaps surprisingly, that the auxiliary tasks alone are sufficient to learn state representations good enough to outperform an end-to-end trained actor-critic baseline. We opensourced our code at https://github.com/Hwhitetooth/random_gvfs.
- 1 July 2021
- Published by Elsevier in Neuropsychologia
- Vol. 158, 107657
Rational use of episodic and working memory: A normative account of prospective memory
- 1 July 2021
- Published by Elsevier in Neuropsychologia
- Vol. 158, 107657
Rational use of episodic and working memory: A normative account of prospective memory
- 1 January 2020
- Published by Cambridge University Press (CUP) in Behavioral and Brain Sciences
- Vol. 43, e14
Cognitively bounded rational analyses and the crucial role of theories of subjective utility
We agree that combining rational analysis with cognitive bounds, what we previously introduced as Cognitively Bounded Rational Analysis, is a promising and under-used methodology in psychology. We further situate the framework in the literature, and highlight the important issue of a theory of subjective utility, which is not addressed sufficiently clearly in the framework or related previous work.
- 1 January 2020
- Published by Cambridge University Press (CUP) in Behavioral and Brain Sciences
- Vol. 43, e14
Cognitively bounded rational analyses and the crucial role of theories of subjective utility
We agree that combining rational analysis with cognitive bounds, what we previously introduced as Cognitively Bounded Rational Analysis, is a promising and under-used methodology in psychology. We further situate the framework in the literature, and highlight the important issue of a theory of subjective utility, which is not addressed sufficiently clearly in the framework or related previous work.
- Open Access
How Should an Agent Practice?
We present a method for learning intrinsic reward functions to drive the learning of an agent during periods of practice in which extrinsic task rewards are not available. During practice, the environment may differ from the one available for training and evaluation with extrinsic rewards. We refer to this setup of alternating periods of practice and objective evaluation as practice-match, drawing an analogy to regimes of skill acquisition common for humans in sports and games. The agent must effectively use periods in the practice environment so that performance improves during matches. In the proposed method the intrinsic practice reward is learned through a meta-gradient approach that adapts the practice reward parameters to reduce the extrinsic match reward loss computed from matches. We illustrate the method on a simple grid world, and evaluate it in two games in which the practice environment differs from match: Pong with practice against a wall without an opponent, and PacMan with practice in a maze without ghosts. The results show gains from learning in practice in addition to match periods over learning in matches only.
Open AccessHow Should an Agent Practice?
We present a method for learning intrinsic reward functions to drive the learning of an agent during periods of practice in which extrinsic task rewards are not available. During practice, the environment may differ from the one available for training and evaluation with extrinsic rewards. We refer to this setup of alternating periods of practice and objective evaluation as practice-match, drawing an analogy to regimes of skill acquisition common for humans in sports and games. The agent must effectively use periods in the practice environment so that performance improves during matches. In the proposed method the intrinsic practice reward is learned through a meta-gradient approach that adapts the practice reward parameters to reduce the extrinsic match reward loss computed from matches. We illustrate the method on a simple grid world, and evaluate it in two games in which the practice environment differs from match: Pong with practice against a wall without an opponent, and PacMan with practice in a maze without ghosts. The results show gains from learning in practice in addition to match periods over learning in matches only.
- Open Access
Rational use of Episodic and Working Memory: A Normative Account of Prospective Memory
A bstract: Humans often simultaneously pursue multiple plans at different time scales. The successful realization of non-immediate plans (e.g., post package after work) requires keeping track of a future plan while accomplishing other intermediate tasks (e.g., write a paper), a capacity known as prospective memory . This capacity requires the integration of noisy evidence from perceptual input with evidence from short-term working memory (WM) and longer-term or episodic memory (LTM/EM). Here we formulate a set of dual-task problems in empirical studies of prospective memory as problems of computational rationality, and ask how a rational model should exploit noisy perception and memory to maximize payoffs. The model combines reinforcement learning (optimal action selection) with evidence accumulation (optimal inference) in order to derive good decision parameters for optimal task performance (i.e., performing an ongoing task while monitoring for a cue that triggers executing a second prospective task). We compare model behavior to key accuracy and reaction time phenomena in human performance. Thus, we offer a normative approach to theorizing and modeling these phenomena without assumptions about mechanisms of attention or retrieval. This approach can be extended to study meta-parameters governing the boundedly rational use of memory in planned action in health, as well as compensatory mnemonic strategies that may be rational responses to disturbances of these mechanisms in neuropsychiatric disorders.
Open AccessRational use of Episodic and Working Memory: A Normative Account of Prospective Memory
A bstract: Humans often simultaneously pursue multiple plans at different time scales. The successful realization of non-immediate plans (e.g., post package after work) requires keeping track of a future plan while accomplishing other intermediate tasks (e.g., write a paper), a capacity known as prospective memory . This capacity requires the integration of noisy evidence from perceptual input with evidence from short-term working memory (WM) and longer-term or episodic memory (LTM/EM). Here we formulate a set of dual-task problems in empirical studies of prospective memory as problems of computational rationality, and ask how a rational model should exploit noisy perception and memory to maximize payoffs. The model combines reinforcement learning (optimal action selection) with evidence accumulation (optimal inference) in order to derive good decision parameters for optimal task performance (i.e., performing an ongoing task while monitoring for a cue that triggers executing a second prospective task). We compare model behavior to key accuracy and reaction time phenomena in human performance. Thus, we offer a normative approach to theorizing and modeling these phenomena without assumptions about mechanisms of attention or retrieval. This approach can be extended to study meta-parameters governing the boundedly rational use of memory in planned action in health, as well as compensatory mnemonic strategies that may be rational responses to disturbances of these mechanisms in neuropsychiatric disorders.
- 1 January 2018
- Published by Cambridge University Press (CUP) in Behavioral and Brain Sciences
- Vol. 41, e232
The role of (bounded) optimization in theory testing and prediction
We argue that a radically increased emphasis on (bounded) optimality can contribute to cognitive science by supporting prediction. Bounded optimality (computational rationality), an idea that borrowed from artificial intelligence, supports a priori behavioral prediction from constrained generative models of cognition. Bounded optimality thereby addresses serious failings with the logic and testing of descriptive models of perception and action.
- 1 January 2018
- Published by Cambridge University Press (CUP) in Behavioral and Brain Sciences
- Vol. 41, e232
The role of (bounded) optimization in theory testing and prediction
We argue that a radically increased emphasis on (bounded) optimality can contribute to cognitive science by supporting prediction. Bounded optimality (computational rationality), an idea that borrowed from artificial intelligence, supports a priori behavioral prediction from constrained generative models of cognition. Bounded optimality thereby addresses serious failings with the logic and testing of descriptive models of perception and action.
- Top Cited Papers
- 20 March 2014
- Published by Wiley in Topics in Cognitive Science
- Vol. 6 (2) , 279-311
Computational Rationality: Linking Mechanism and Behavior Through Bounded Utility Maximization
We propose a framework for including information‐processing bounds in rational analyses. It is an application of bounded optimality (Russell & Subramanian, 1995) to the challenges of developing theories of mechanism and behavior. The framework is based on the idea that behaviors are generated by cognitive mechanisms that are adapted to the structure of not only the environment but also the mind and brain itself. We call the framework computational rationality to emphasize the incorporation of computational mechanism into the definition of rational action. Theories are specified as optimal program problems, defined by an adaptation environment, a bounded machine, and a utility function. Such theories yield different classes of explanation, depending on the extent to which they emphasize adaptation to bounds, and adaptation to some ecology that differs from the immediate local environment. We illustrate this variation with examples from three domains: visual attention in a linguistic task, manual response ordering, and reasoning. We explore the relation of this framework to existing “levels” approaches to explanation, and to other optimality‐based modeling approaches.
- 20 March 2014
- Published by Wiley in Topics in Cognitive Science
- Vol. 6 (2) , 279-311
Top Cited PapersComputational Rationality: Linking Mechanism and Behavior Through Bounded Utility Maximization
We propose a framework for including information‐processing bounds in rational analyses. It is an application of bounded optimality (Russell & Subramanian, 1995) to the challenges of developing theories of mechanism and behavior. The framework is based on the idea that behaviors are generated by cognitive mechanisms that are adapted to the structure of not only the environment but also the mind and brain itself. We call the framework computational rationality to emphasize the incorporation of computational mechanism into the definition of rational action. Theories are specified as optimal program problems, defined by an adaptation environment, a bounded machine, and a utility function. Such theories yield different classes of explanation, depending on the extent to which they emphasize adaptation to bounds, and adaptation to some ecology that differs from the immediate local environment. We illustrate this variation with examples from three domains: visual attention in a linguistic task, manual response ordering, and reasoning. We explore the relation of this framework to existing “levels” approaches to explanation, and to other optimality‐based modeling approaches.
- Top Cited Papers
- 1 October 2006
- Published by Elsevier in Trends in Cognitive Sciences
- Vol. 10 (10) , 447-454
Computational principles of working memory in sentence comprehension
- 1 October 2006
- Published by Elsevier in Trends in Cognitive Sciences
- Vol. 10 (10) , 447-454
Top Cited PapersComputational principles of working memory in sentence comprehension