On The Stability of Moral Preferences: A Problem with Computational Elicitation Methods
Open Access
- 16 October 2024
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
- Vol. 7, 156-167
Abstract
Preference elicitation frameworks feature heavily in the research on participatory ethical AI tools and provide a viable mechanism to enquire and incorporate the moral values of various stakeholders. As part of the elicitation process, surveys about moral preferences, opinions, and judgments are typically administered only once to each participant. This methodological practice is reasonable if participants’ responses are stable over time such that, all other things being held constant, their responses today will be the same as their responses to the same questions at a later time. However, we do not know how often that is the case. It is possible that participants’ true moral preferences change, are subject to temporary moods or whims, or are influenced by environmental factors we don’t track. If participants’ moral responses are unstable in such ways, it would raise important methodological and theoretical issues for how participants’ true moral preferences, opinions, and judgments can be ascertained. We address this possibility here by asking the same survey participants the same moral questions about which patient should receive a kidney when only one is available ten times in ten different sessions over two weeks, varying only presentation order across sessions. We measured how often participants gave different responses to simple (Study One) and more complicated (Study Two) controversial and uncontroversial repeated scenarios. On average, the fraction of times participants changed their responses to controversial scenarios (i.e., were unstable) was around 10-18% (±14-15%) across studies, and this instability is observed to have positive associations with response time and decision-making difficulty. We discuss the implications of these results for the efficacy of common moral preference elicitation methods, highlighting the role of response instability in potentially causing value misalignment between the stakeholders and AI tools trained on their moral judgments.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Value Preferences Estimation and Disambiguation in Hybrid Participatory SystemsJournal of Artificial Intelligence Research, 2025
- Preferences as fairness judgments: a critical review of normative frameworks of preference elicitation and development of an alternative based on constitutional economicsCost Effectiveness and Resource Allocation, 2024
- How Stable are Moral Judgments?Review of Philosophy and Psychology, 2022
- Adapting a kidney exchange algorithm to align with human valuesArtificial Intelligence, 2020
- Crowdsourcing moral machinesCommunications of the ACM, 2020
- Mapping morality with a compass: Testing the theory of ‘morality-as-cooperation’ with a new questionnaireJournal of Research in Personality, 2019
- The Moral Machine experimentNature, 2018
- Assessing Test-Retest Reliability of Psychological MeasuresEuropean Psychologist, 2017
- Ordinal preference elicitation methods in health economics and health services research: using discrete choice experiments and ranking methodsBritish Medical Bulletin, 2012
- Interrater reliability: the kappa statisticBiochemia Medica, 2012