Leveraging Implicit Sentiments: Enhancing Reliability and Validity in Psychological Trait Evaluation of LLMs
Preprint
Abstract
Recent advancements in Large Language Models (LLMs) have led to their increasing integration into human life. With the transition from mere tools to human-like assistants, understanding their psychological aspects-such as emotional tendencies and personalities-becomes essential for ensuring their trustworthiness. However, current psychological evaluations of LLMs, often based on human psychological assessments like the BFI, face significant limitations. The results from these approaches often lack reliability and have limited validity when predicting LLM behavior in real-world scenarios. In this work, we introduce a novel evaluation instrument specifically designed for LLMs, called Core Sentiment Inventory (CSI). CSI is a bilingual tool, covering both English and Chinese, that implicitly evaluates models' sentiment tendencies, providing an insightful psychological portrait of LLM across three dimensions: optimism, pessimism, and neutrality. Through extensive experiments, we demonstrate that: 1) CSI effectively captures nuanced emotional patterns, revealing significant variation in LLMs across languages and contexts; 2) Compared to current approaches, CSI significantly improves reliability, yielding more consistent results; and 3) The correlation between CSI scores and the sentiment of LLM's real-world outputs exceeds 0.85, demonstrating its strong validity in predicting LLM behavior. We make CSI public available via: https://github.com/dependentsign/CSI.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- A psychometric framework for evaluating and shaping personality traits in large language modelsNature Machine Intelligence, 2025
- Explicitly unbiased large language models still form biased associationsProceedings of the National Academy of Sciences of the United States of America, 2025
- Opportunities and risks of large language models in psychiatryNPP—Digital Psychiatry and Neuroscience, 2024
- Large language models could change the future of behavioral healthcare: a proposal for responsible development and evaluationnpj Mental Health Research, 2024
- InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews2024
- Toward a dimensional model of vocational interests.Journal of Applied Psychology, 2019
- Relationship Between Emotional Intelligence and Health Behaviours among University Students: The Predictive and Moderating Role of GenderBioMed Research International, 2018
- Enhancing Cultural Intelligence: The Roles of Implicit Culture Beliefs and AdjustmentPersonnel Psychology, 2016
- Wage Cuts and Managers’ Empathy: How a Positive Emotion Can Contribute to Positive Organizational Ethics in Difficult TimesJournal of Business Ethics, 2013
- Development and validation of a measure of emotional intelligencePersonality and Individual Differences, 1998