Evaluating Moral Beliefs across LLMs through a Pluralistic Framework
Preprint
Abstract
Proper moral beliefs are fundamental for language models, yet assessing these beliefs poses a significant challenge. This study introduces a novel three-module framework to evaluate the moral beliefs of four prominent large language models. Initially, we constructed a dataset containing 472 moral choice scenarios in Chinese, derived from moral words. The decision-making process of the models in these scenarios reveals their moral principle preferences. By ranking these moral choices, we discern the varying moral beliefs held by different language models. Additionally, through moral debates, we investigate the firmness of these models to their moral choices. Our findings indicate that English language models, namely ChatGPT and Gemini, closely mirror moral decisions of the sample of Chinese university students, demonstrating strong adherence to their choices and a preference for individualistic moral beliefs. In contrast, Chinese models such as Ernie and ChatGLM lean towards collectivist moral beliefs, exhibiting ambiguity in their moral choices and debates. This study also uncovers gender bias embedded within the moral beliefs of all examined language models. Our methodology offers an innovative means to assess moral beliefs in both artificial and human intelligence, facilitating a comparison of moral values across different cultures.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates2024
- ChatGPT outperforms crowd workers for text-annotation tasksProceedings of the National Academy of Sciences, 2023
- Occupational gender segregation and gendered language in a language without gender: trends, variations, implications for social development in ChinaHumanities and Social Sciences Communications, 2021
- High and low conflict moral dilemmas resolution: comparing moral judgment from Spanish and Mexican samplesAustralian Journal of Psychology, 2021
- The comparison of the wisdom view in Chinese and Western culturesCurrent Psychology, 2021
- Moral identity in cultural context: Differences between Canadian and Chinese university studentsJournal of Moral Education, 2018
- Consequences, norms, and generalized inaction in moral dilemmas: The CNI model of moral decision-making.Journal of Personality and Social Psychology, 2017
- Cultural differences in moral judgment and behavior, across and within societiesCurrent Opinion in Psychology, 2016
- Functional and clinical neuroanatomy of moralityBrain, 2012
- Liberals and conservatives rely on different sets of moral foundations.Journal of Personality and Social Psychology, 2009