Can LLMs Faithfully Explain Themselves in Low-Resource Languages? A Case Study on Emotion Detection in Persian
Preprint
Abstract
Large language models (LLMs) are increasingly used to generate self-explanations alongside their predictions, a practice that raises concerns about the faithfulness of these explanations, especially in low-resource languages. This study evaluates the faithfulness of LLM-generated explanations in the context of emotion classification in Persian, a low-resource language, by comparing the influential words identified by the model against those identified by human annotators. We assess faithfulness using confidence scores derived from token-level log-probabilities. Two prompting strategies, differing in the order of explanation and prediction (Predict-then-Explain and Explain-then-Predict), are tested for their impact on explanation faithfulness. Our results reveal that while LLMs achieve strong classification performance, their generated explanations often diverge from faithful reasoning, showing greater agreement with each other than with human judgments. These results highlight the limitations of current explanation methods and metrics, emphasizing the need for more robust approaches to ensure LLM reliability in multilingual and low-resource contexts.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- ArmanEmo: a Persian dataset for text-based emotion detectionLanguage Resources and Evaluation, 2025
- Evaluating the Reliability of Self-explanations in Large Language Models2025
- Comparing human text classification performance and explainability with large language and machine learning models using eye-trackingScientific Reports, 2024
- TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation2024
- Are self-explanations from Large Language Models faithful?2024
- Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings2024
- Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification2021
- Fooling LIME and SHAP2020
- ERASER: A Benchmark to Evaluate Rationalized NLP Models2020
- "Why Should I Trust You?"2016