ExpertQA: Expert-Curated Questions and Attributed Answers
Preprint
Abstract
As language models are adopted by a more sophisticated and diverse set of users, the importance of guaranteeing that they provide factually correct information supported by verifiable sources is critical across fields of study. This is especially the case for high-stakes fields, such as medicine and law, where the risk of propagating false information is high and can lead to undesirable societal consequences. Previous work studying attribution and factuality has not focused on analyzing these characteristics of language model outputs in domain-specific scenarios. In this work, we conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality, by bringing domain experts in the loop. Specifically, we collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. In addition, we ask experts to improve upon responses from language models. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Science in the age of large language modelsNature Reviews Physics, 2023
- Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for MedicineNew England Journal of Medicine, 2023
- How Nature readers are using ChatGPTNature, 2023
- On scientific understanding with artificial intelligenceNature Reviews Physics, 2022
- Evaluating Attribution in Dialogue Systems: The BEGIN BenchmarkTransactions of the Association for Computational Linguistics, 2022
- Fact Checking with Insufficient EvidenceTransactions of the Association for Computational Linguistics, 2022
- CoQA: A Conversational Question Answering ChallengeTransactions of the Association for Computational Linguistics, 2019
- An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competitionBMC Bioinformatics, 2015
- The Probabilistic Relevance Framework: BM25 and BeyondFoundations and Trends® in Information Retrieval, 2009
- Understanding user goals in web search2004