The Capacity for Moral Self-Correction in Large Language Models
Authors:
- Deep GanguliDGDeep Ganguli
- Amanda AskellAAAmanda Askell
- Nicholas SchieferNSNicholas Schiefer
- Thomas I. LiaoTLThomas I. Liao
- Kamilė LukošiūtėKLKamilė Lukošiūtė
- Anna ChenACAnna Chen
- Anna GoldieAGAnna Goldie
- Azalia MirhoseiniAMAzalia Mirhoseini
- Catherine OlssonCOCatherine Olsson
- Danny HernandezDHDanny Hernandez
- Dawn DrainDDDawn Drain
- Dustin LiDLDustin Li
Top Cited Papers
Preprint
Abstract
We test the hypothesis that language models trained with reinforcement learning from human feedback (RLHF) have the capability to "morally self-correct" -- to avoid producing harmful outputs -- if instructed to do so. We find strong evidence in support of this hypothesis across three different experiments, each of which reveal different facets of moral self-correction. We find that the capability for moral self-correction emerges at 22B model parameters, and typically improves with increasing model size and RLHF training. We believe that at this level of scale, language models obtain two capabilities that they can use for moral self-correction: (1) they can follow instructions and (2) they can learn complex normative concepts of harm like stereotyping, bias, and discrimination. As such, they can follow instructions to avoid certain kinds of morally harmful outputs. We believe our results are cause for cautious optimism regarding the ability to train language models to abide by ethical principles.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Counterfactual Fairness Is Basically Demographic ParityProceedings of the AAAI Conference on Artificial Intelligence, 2023
- Inverse Scaling Can Become U-Shaped2023
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them2023
- Predictability and Surprise in Large Generative Models2022
- An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models2022
- Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPTransactions of the Association for Computational Linguistics, 2021
- Large language models associate Muslims with violenceNature Machine Intelligence, 2021
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models2020
- Reducing Sentiment Bias in Language Models via Counterfactual Evaluation2020
- Social Bias Frames: Reasoning about Social and Power Implications of Language2020