CARE-Bench: A Benchmark of Diverse Client Simulations Guided by Expert Principles for Evaluating LLMs in Psychological Counseling
Preprint
Abstract
The mismatch between the growing demand for psychological counseling and the limited availability of services has motivated research into the application of Large Language Models (LLMs) in this domain. Consequently, there is a need for a robust and unified benchmark to assess the counseling competence of various LLMs. Existing works, however, are limited by unprofessional client simulation, static question-and-answer evaluation formats, and unidimensional metrics. These limitations hinder their effectiveness in assessing a model's comprehensive ability to handle diverse and complex clients. To address this gap, we introduce \textbf{CARE-Bench}, a dynamic and interactive automated benchmark. It is built upon diverse client profiles derived from real-world counseling cases and simulated according to expert guidelines. CARE-Bench provides a multidimensional performance evaluation grounded in established psychological scales. Using CARE-Bench, we evaluate several general-purpose LLMs and specialized counseling models, revealing their current limitations. In collaboration with psychologists, we conduct a detailed analysis of the reasons for LLMs' failures when interacting with clients of different types, which provides directions for developing more comprehensive, universal, and effective counseling models.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- A meta-analytic review of personality traits and their associations with mental health treatment outcomesClinical Psychology Review, 2019
- The Counseling Competencies Scale: Validation and RefinementMeasurement and Evaluation in Counseling and Development, 2017
- Does personality predict health and well-being? A metasynthesis.Health Psychology, 2017
- Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled TrialJMIR Mental Health, 2017
- What works for whom: Tailoring psychotherapy to the personJournal of Clinical Psychology, 2010
- Linking “big” personality traits to anxiety, depressive, and substance use disorders: A meta-analysis.Psychological Bulletin, 2010
- The Relationship Between the Five-Factor Model of Personality and Symptoms of Clinical Disorders: A Meta-AnalysisJournal of Psychopathology and Behavioral Assessment, 2005