InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
Preprint
Abstract
Role-playing agents (RPAs), powered by large language models, have emerged as a flourishing field of applications. However, a key challenge lies in assessing whether RPAs accurately reproduce the personas of target characters, namely their character fidelity. Existing methods mainly focus on the knowledge and linguistic patterns of characters. This paper, instead, introduces a novel perspective to evaluate the personality fidelity of RPAs with psychological scales. Overcoming drawbacks of previous self-report assessments on RPAs, we propose InCharacter, namely Interviewing Character agents for personality tests. Experiments include various types of RPAs and LLMs, covering 32 distinct characters on 14 widely used psychological scales. The results validate the effectiveness of InCharacter in measuring RPA personalities. Then, with InCharacter, we show that state-of-the-art RPAs exhibit personalities highly aligned with the human-perceived personalities of the characters, achieving an accuracy up to 80.7%. Our demo, code, dataset, and results are publicly available at https://github.com/Neph0s/InCharacter.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- A psychometric framework for evaluating and shaping personality traits in large language modelsNature Machine Intelligence, 2025
- The Self-Perception and Political Biases of ChatGPTHuman Behavior and Emerging Technologies, 2024
- On the Reliability of Psychological Scales on Large Language Models2024
- RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models2024
- CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation2024
- LiveChat: A Large-Scale Personalized Dialogue Dataset Automatically Constructed from Live Streaming2023
- Who is GPT-3? An exploration of personality, values and demographics2022
- SELF-REPORT AND CLINICIAN-RATED MEASURES OF DEPRESSION SEVERITY: CAN ONE REPLACE THE OTHER?Depression and Anxiety, 2012
- Self-reported versus clinician-rated symptoms of depression as outcome measures in psychotherapy research on depression: A meta-analysisClinical Psychology Review, 2010
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968