Training Value-Aligned Reinforcement Learning Agents Using a Normative Prior
- 16 February 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 5 (7) , 3350-3361
Abstract
Value alignment is a property of intelligent agents wherein they solely pursue non-harmful behaviors or human-beneficial goals. We introduce an approach to value-aligned reinforcement learning, in which we train an agent with two reward signals: a standard task performance reward plus a normative behavior reward. The normative behavior reward is derived from a value-aligned prior model that we train using naturally occurring stories. These stories encode societal norms and can be used to classify text as normative or non-normative. We show how variations on a policy shaping technique can balance these two sources of reward and produce policies that are both effective and perceived as more normative. We test our value-alignment technique on three interactive text-based worlds; each world is designed specifically to challenge agents with a task as well as provide opportunities to deviate from the task to engage in normative and/or altruistic behavior.Keywords
All Related Versions
Funding Information
- National Science Foundation (1849231)
This publication has 13 references indexed in Scilit:
- Artificial Intelligence and the Problem of Control2021
- Story Realization: Expanding Plot Events into SentencesProceedings of the AAAI Conference on Artificial Intelligence, 2020
- TextWorld: A Learning Environment for Text-Based Games2019
- Efficient Supervision for Robot Learning Via Imitation, Simulation, and AdaptationKI - Künstliche Intelligenz, 2019
- Policy Shaping with Supervisory Attention Driven Exploration2018
- Transfer Learning for Low-Resource Neural Machine Translation2016
- Research Priorities for Robust and Beneficial Artificial IntelligenceAI Magazine, 2015
- APRIL: Active Preference Learning-Based Reinforcement Learning2012
- The Nature, Importance, and Difficulty of Machine EthicsIEEE Intelligent Systems, 2006
- Apprenticeship learning via inverse reinforcement learning2004