Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family
Preprint
Abstract
ChatGPT is a powerful large language model (LLM) that covers knowledge resources such as Wikipedia and supports natural language question answering using its own knowledge. Therefore, there is growing interest in exploring whether ChatGPT can replace traditional knowledge-based question answering (KBQA) models. Although there have been some works analyzing the question answering performance of ChatGPT, there is still a lack of large-scale, comprehensive testing of various types of complex questions to analyze the limitations of the model. In this paper, we present a framework that follows the black-box testing specifications of CheckList proposed by Ribeiro et. al. We evaluate ChatGPT and its family of LLMs on eight real-world KB-based complex question answering datasets, which include six English datasets and two multilingual datasets. The total number of test cases is approximately 190,000. In addition to the GPT family of LLMs, we also evaluate the well-known FLAN-T5 to identify commonalities between the GPT family and other LLMs. The dataset and code are available at https://github.com/tan92hl/Complex-Question-Answering-Evaluation-of-GPT-family.gitKeywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Uniqorn: Unified question answering over RDF knowledge graphs and natural language textJournal of Web Semantics, 2024
- PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts2023
- Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?2023
- ChatGPT: Jack of all trades, master of noneInformation Fusion, 2023
- Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech2023
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity2023
- Is ChatGPT a General-Purpose Natural Language Processing Task Solver?2023
- MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question AnsweringTransactions of the Association for Computational Linguistics, 2021
- How Can We Know What Language Models Know?Transactions of the Association for Computational Linguistics, 2020
- Analysis Methods in Neural Language Processing: A SurveyTransactions of the Association for Computational Linguistics, 2019