Sheila A. McIlraith
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Sheila A. McIlraith has a total of 55 co-authors. Here are the top co-authors by number of shared publications:
Toryn Q. Klassen
University of Toronto, Toronto, Canada
6 shared publications- Christian Muise
Queen's University Kingston Canada
4 shared publications - Diane Horton
Department of Computer Science, University of Toronto, Toronto, Ontario, Canada
4 shared publications - Andrew C. Li
Department of Computer Science Institute for Artificial Intelligence University of Toronto Vector Schwartz Reisman Institute for Technology and Society Toronto Canada
3 shared publications - Eldan Cohen
University of Toronto, Department of Mechanical and Industrial Engineering, Toronto, Canada
3 shared publications - Maayan Shvo
University of Toronto
3 shared publications
Latest Publications
- Open Access
- 25 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 4162-4198
Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems
- 25 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 4162-4198
Open AccessFormal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems
- Open Access
- 1 November 2025
- Published by International Joint Conferences on Artificial Intelligence
- Vol. 22 (1) , 566-576
Pushdown Reward Machines for Reinforcement Learning
Reward machines (RMs) are automata structures that encode (non-Markovian) reward functions for reinforcement learning (RL). RMs can reward any behaviour representable in regular languages and, when paired with RL algorithms that exploit RM structure, have been shown to significantly improve sample efficiency in many domains. In this work, we present pushdown reward machines (pdRMs), an extension of reward machines based on deterministic pushdown automata. pdRMs can recognise and reward temporally extended behaviours representable in deterministic context-free languages, making them more expressive than reward machines. We introduce two variants of pdRM-based policies, one which has access to the entire stack of the pdRM, and one which can only access the top k symbols (for a given constant k) of the stack. We propose a procedure to check when the two kinds of policies (for a given environment, pdRM, and constant k) achieve the same optimal state values. We then provide theoretical results establishing the expressive power of pdRMs, and space complexity results for the proposed learning problems. Lastly, we propose an approach for off-policy RL algorithms that exploits counterfactual experiences with pdRMs. We conclude by providing experimental results showing how agents can be trained to perform tasks representable in deterministic context-free languages using pdRMs.
- 1 November 2025
- Published by International Joint Conferences on Artificial Intelligence
- Vol. 22 (1) , 566-576
Open AccessPushdown Reward Machines for Reinforcement Learning
Reward machines (RMs) are automata structures that encode (non-Markovian) reward functions for reinforcement learning (RL). RMs can reward any behaviour representable in regular languages and, when paired with RL algorithms that exploit RM structure, have been shown to significantly improve sample efficiency in many domains. In this work, we present pushdown reward machines (pdRMs), an extension of reward machines based on deterministic pushdown automata. pdRMs can recognise and reward temporally extended behaviours representable in deterministic context-free languages, making them more expressive than reward machines. We introduce two variants of pdRM-based policies, one which has access to the entire stack of the pdRM, and one which can only access the top k symbols (for a given constant k) of the stack. We propose a procedure to check when the two kinds of policies (for a given environment, pdRM, and constant k) achieve the same optimal state values. We then provide theoretical results establishing the expressive power of pdRMs, and space complexity results for the proposed learning problems. Lastly, we propose an approach for off-policy RL algorithms that exploits counterfactual experiences with pdRMs. We conclude by providing experimental results showing how agents can be trained to perform tasks representable in deterministic context-free languages using pdRMs.
- Open Access
Pluralistic Alignment Over Time
If an AI system makes decisions over time, how should we evaluate how aligned it is with a group of stakeholders (who may have conflicting values and preferences)? In this position paper, we advocate for consideration of temporal aspects including stakeholders' changing levels of satisfaction and their possibly temporally extended preferences. We suggest how a recent approach to evaluating fairness over time could be applied to a new form of pluralistic alignment: temporal pluralism, where the AI system reflects different stakeholders' values at different times.
Open AccessPluralistic Alignment Over Time
If an AI system makes decisions over time, how should we evaluate how aligned it is with a group of stakeholders (who may have conflicting values and preferences)? In this position paper, we advocate for consideration of temporal aspects including stakeholders' changing levels of satisfaction and their possibly temporally extended preferences. We suggest how a recent approach to evaluating fairness over time could be applied to a new form of pluralistic alignment: temporal pluralism, where the AI system reflects different stakeholders' values at different times.
- Open Access
Being Considerate as a Pathway Towards Pluralistic Alignment for Agentic AI
Pluralistic alignment is concerned with ensuring that an AI system's objectives and behaviors are in harmony with the diversity of human values and perspectives. In this paper we study the notion of pluralistic alignment in the context of agentic AI, and in particular in the context of an agent that is trying to learn a policy in a manner that is mindful of the values and perspective of others in the environment. To this end, we show how being considerate of the future wellbeing and agency of other (human) agents can promote a form of pluralistic alignment.
Open AccessBeing Considerate as a Pathway Towards Pluralistic Alignment for Agentic AI
Pluralistic alignment is concerned with ensuring that an AI system's objectives and behaviors are in harmony with the diversity of human values and perspectives. In this paper we study the notion of pluralistic alignment in the context of agentic AI, and in particular in the context of an agent that is trying to learn a policy in a manner that is mindful of the values and perspective of others in the environment. To this end, we show how being considerate of the future wellbeing and agency of other (human) agents can promote a form of pluralistic alignment.
- Open Access
- 25 June 2024
- Published by Association for Computing Machinery (ACM)
Embedded Ethics: Pandemic Contact Tracing and Ethical Trade-offs
- 25 June 2024
- Published by Association for Computing Machinery (ACM)
Open AccessEmbedded Ethics: Pandemic Contact Tracing and Ethical Trade-offs
- Open Access
- 24 June 2024
- Published by Association for Computing Machinery (ACM)
Embedded Ethics: Pandemic Exposure Notification Systems and Giving Ethical Justifications
- 24 June 2024
- Published by Association for Computing Machinery (ACM)
Open AccessEmbedded Ethics: Pandemic Exposure Notification Systems and Giving Ethical Justifications
- Open Access
- 24 March 2024
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (18) , 20212-20221
PRP Rebooted: Advancing the State of the Art in FOND Planning
Fully Observable Non-Deterministic (FOND) planning is a variant of classical symbolic planning in which actions are nondeterministic, with an action's outcome known only upon execution. It is a popular planning paradigm with applications ranging from robot planning to dialogue-agent design and reactive synthesis. Over the last 20 years, a number of approaches to FOND planning have emerged. In this work, we establish a new state of the art, following in the footsteps of some of the most powerful FOND planners to date. Our planner, PR2, decisively outperforms the four leading FOND planners, at times by a large margin, in 17 of 18 domains that represent a comprehensive benchmark suite. Ablation studies demonstrate the impact of various techniques we introduce, with the largest improvement coming from our novel FOND-aware heuristic.
- 24 March 2024
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (18) , 20212-20221
Open AccessPRP Rebooted: Advancing the State of the Art in FOND Planning
Fully Observable Non-Deterministic (FOND) planning is a variant of classical symbolic planning in which actions are nondeterministic, with an action's outcome known only upon execution. It is a popular planning paradigm with applications ranging from robot planning to dialogue-agent design and reactive synthesis. Over the last 20 years, a number of approaches to FOND planning have emerged. In this work, we establish a new state of the art, following in the footsteps of some of the most powerful FOND planners to date. Our planner, PR2, decisively outperforms the four leading FOND planners, at times by a large margin, in 17 of 18 domains that represent a comprehensive benchmark suite. Ablation studies demonstrate the impact of various techniques we introduce, with the largest improvement coming from our novel FOND-aware heuristic.
- Open Access
- 7 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 533-539
Do Embedded Ethics Modules Have Impact Beyond the Classroom?
Embedded ethics education integrates ethical considerations into computer sciences courses in support of ethics-informed design, development, and deployment of technology. Scholarly assessment has demonstrated that such modules can influence students' attitudes about the relevance and importance of ethics to their work, as well as their perceived ability to tackle ethical issues in the workplace. In this paper, we report on a study that investigates whether embedded ethics modules have an impact beyond the classroom. Specifically, we examine whether embedded ethics modules influence students to learn more about ethics on their own, whether students are better able to recognize ethical issues when they enter the workplace for an industrial or research work experience, and whether they report that the modules they participated in helped them to navigate the ethical situations they encountered at work. While further assessment is needed to investigate these questions fully, our results suggest that embedded ethics modules can indeed have this kind of positive impact beyond the classroom.
- 7 March 2024
- Published by Association for Computing Machinery (ACM)
- p. 533-539
Open AccessDo Embedded Ethics Modules Have Impact Beyond the Classroom?
Embedded ethics education integrates ethical considerations into computer sciences courses in support of ethics-informed design, development, and deployment of technology. Scholarly assessment has demonstrated that such modules can influence students' attitudes about the relevance and importance of ethics to their work, as well as their perceived ability to tackle ethical issues in the workplace. In this paper, we report on a study that investigates whether embedded ethics modules have an impact beyond the classroom. Specifically, we examine whether embedded ethics modules influence students to learn more about ethics on their own, whether students are better able to recognize ethical issues when they enter the workplace for an industrial or research work experience, and whether they report that the modules they participated in helped them to navigate the ethical situations they encountered at work. While further assessment is needed to investigate these questions fully, our results suggest that embedded ethics modules can indeed have this kind of positive impact beyond the classroom.
- Open Access
STEVE-1: A Generative Model for Text-to-Behavior in Minecraft
Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces a methodology, inspired by unCLIP, for instruction-tuning generative models of behavior without relying on a large dataset of instruction-labeled trajectories. Using this methodology, we create an instruction-tuned Video Pretraining (VPT) model called STEVE-1, which can follow short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, reducing the need for costly human text annotations, and all for only $60 of compute. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 sets a new bar for open-ended instruction-following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.
Open AccessSTEVE-1: A Generative Model for Text-to-Behavior in Minecraft
Constructing AI models that respond to text instructions is challenging, especially for sequential decision-making tasks. This work introduces a methodology, inspired by unCLIP, for instruction-tuning generative models of behavior without relying on a large dataset of instruction-labeled trajectories. Using this methodology, we create an instruction-tuned Video Pretraining (VPT) model called STEVE-1, which can follow short-horizon open-ended text and visual instructions in Minecraft. STEVE-1 is trained in two steps: adapting the pretrained VPT model to follow commands in MineCLIP's latent space, then training a prior to predict latent codes from text. This allows us to finetune VPT through self-supervised behavioral cloning and hindsight relabeling, reducing the need for costly human text annotations, and all for only $60 of compute. By leveraging pretrained models like VPT and MineCLIP and employing best practices from text-conditioned image generation, STEVE-1 sets a new bar for open-ended instruction-following in Minecraft with low-level controls (mouse and keyboard) and raw pixel inputs, far outperforming previous baselines and robustly completing 12 of 13 tasks in our early-game evaluation suite. We provide experimental evidence highlighting key factors for downstream performance, including pretraining, classifier-free guidance, and data scaling. All resources, including our model weights, training scripts, and evaluation tools are made available for further research.
- Open Access
PRP Rebooted: Advancing the State of the Art in FOND Planning
Fully Observable Non-Deterministic (FOND) planning is a variant of classical symbolic planning in which actions are nondeterministic, with an action's outcome known only upon execution. It is a popular planning paradigm with applications ranging from robot planning to dialogue-agent design and reactive synthesis. Over the last 20 years, a number of approaches to FOND planning have emerged. In this work, we establish a new state of the art, following in the footsteps of some of the most powerful FOND planners to date. Our planner, PR2, decisively outperforms the four leading FOND planners, at times by a large margin, in 17 of 18 domains that represent a comprehensive benchmark suite. Ablation studies demonstrate the impact of various techniques we introduce, with the largest improvement coming from our novel FOND-aware heuristic.
Open AccessPRP Rebooted: Advancing the State of the Art in FOND Planning
Fully Observable Non-Deterministic (FOND) planning is a variant of classical symbolic planning in which actions are nondeterministic, with an action's outcome known only upon execution. It is a popular planning paradigm with applications ranging from robot planning to dialogue-agent design and reactive synthesis. Over the last 20 years, a number of approaches to FOND planning have emerged. In this work, we establish a new state of the art, following in the footsteps of some of the most powerful FOND planners to date. Our planner, PR2, decisively outperforms the four leading FOND planners, at times by a large margin, in 17 of 18 domains that represent a comprehensive benchmark suite. Ablation studies demonstrate the impact of various techniques we introduce, with the largest improvement coming from our novel FOND-aware heuristic.