Large Language Models as Evaluators for Recommendation Explanations
Open Access
- 8 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 33-42
Abstract
The explainability of recommender systems has attracted significant attention in academia and industry. Many efforts have been made for explainable recommendations, yet evaluating the quality of the explanations remains a challenging and unresolved issue. In recent years, leveraging LLMs as evaluators presents a promising avenue in Natural Language Processing tasks (e.g., sentiment classification, information extraction), as they perform strong capabilities in instruction following and common-sense reasoning. However, evaluating recommendation explanatory texts is different from these NLG tasks, as its criteria are related to human perceptions and are usually subjective. In this paper, we investigate whether LLMs can serve as evaluators of recommendation explanations. To answer the question, we utilize real user feedback on explanations given from previous work and additionally collect third-party annotations and LLM evaluations. We design and apply a 3-level meta-evaluation strategy to measure the correlation between evaluator labels and the ground truth provided by users. Our experiments reveal that LLMs, such as GPT4, can provide comparable evaluations with appropriate prompts and settings. We also provide further insights into combining human labels with the LLM evaluation process and utilizing ensembles of multiple heterogeneous LLM evaluators to enhance the accuracy and stability of evaluations. Our study verifies that utilizing LLMs as evaluators can be an accurate, reproducible and cost-effective solution for evaluating recommendation explanation texts. Our code is available here1.Keywords
All Related Versions
Funding Information
- Natural Science Foundation of China (U21B2026)
- Natural Science Foundation of China (62372260)
This publication has 18 references indexed in Scilit:
- Collaborative-Enhanced Prediction of Spending on Newly Downloaded Mobile Games under Consumption Uncertainty2024
- A Reusable Model-agnostic Framework for Faithfully Explainable Recommendation and System ScrutabilityACM Transactions on Information Systems, 2023
- ChatGPT outperforms crowd workers for text-annotation tasksProceedings of the National Academy of Sciences, 2023
- User Perception of Recommendation Explanation: Are Your Explanations What Users Need?ACM Transactions on Information Systems, 2023
- ExpScore: Learning Metrics for Recommendation Explanation2022
- An Unsupervised Aspect-Aware Recommendation Model with Explanation Text GenerationACM Transactions on Information Systems, 2021
- EX3: Explainable Attribute-aware Item-set Recommendations2021
- Measuring Recommendation Explanation Quality2020
- Dual Learning for Explainable Recommendation: Towards Unifying User Preference Prediction and Review Generation2020
- Factorization Machines2010