Does Role-Playing Chatbots Capture the Character Personalities? Assessing Personality Traits for Role-Playing Chatbots
Authors:
Preprint
Abstract
The emergence of large-scale pretrained language models has revolutionized the capabilities of new AI application, especially in the realm of crafting chatbots with distinct personas. Given the "stimulus-response" nature of chatbots, this paper unveils an innovative open-ended interview-style approach for personality assessment on role-playing chatbots, which offers a richer comprehension of their intrinsic personalities. We conduct personality assessments on 32 role-playing chatbots created by the ChatHaruhi library, across both the Big Five and MBTI dimensions, and measure their alignment with human perception. Evaluation results underscore that modern role-playing chatbots based on LLMs can effectively portray personality traits of corresponding characters, with an alignment rate of 82.8% compared with human-perceived personalities. Besides, we also suggest potential strategies for shaping chatbots' personalities. Hence, this paper serves as a cornerstone study for role-playing chatbots that intersects computational linguistics and psychology. Our resources are available at https://github.com/LC1332/Chat-Haruhi-SuzumiyaKeywords
All Related Versions
This publication has 6 references indexed in Scilit:
- A psychometric framework for evaluating and shaping personality traits in large language modelsNature Machine Intelligence, 2025
- The Self-Perception and Political Biases of ChatGPTHuman Behavior and Emerging Technologies, 2024
- Can ChatGPT Assess Human Personalities? A General Evaluation Framework2023
- MPCHAT: Towards Multimodal Persona-Grounded Conversation2023
- LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streaming2023
- Who is GPT-3? An exploration of personality, values and demographics2022