Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answering
Top Cited Papers
Open Access
- 1 January 2025
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 18477-18494
Abstract
Francesco Maria Molfese, Luca Moroni, Luca Gioffré, Alessandro Scirè, Simone Conia, Roberto Navigli. Findings of the Association for Computational Linguistics: ACL 2025. 2025.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Truth or Mirage? 🏝 Towards End-To-End Factuality Evaluation with LLM-O asisComputational Linguistics, 2026
- Do Large Language Models have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs2025
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark2024
- ZEBRA: Zero-Shot Example-Based Retrieval Augmentation for Commonsense Question Answering2024
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards2024
- “My Answer is C”: First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models2024
- Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?2024
- What’s the Meaning of Superhuman Performance in Today’s NLU?2023
- CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge2019
- Social IQa: Commonsense Reasoning about Social Interactions2019