KQA Pro: A Large-Scale Dataset with Interpretable Programs and Accurate SPARQLs for Complex Question Answering over Knowledge Base
Authors:
- Jiaxin ShiJSJiaxin Shi
- Shulin CaoSCShulin Cao
- Liangming PanLPLiangming Pan
- Yutong XiangYXYutong Xiang
- Lei HouLHLei Hou
- Juanzi LiJLJuanzi Li
- Hanwang ZhangHZHanwang Zhang
- Bin HeBHBin He
Preprint
Abstract
Complex question answering over knowledge base (Complex KBQA) is challenging because it requires various compositional reasoning capabilities, such as multi-hop inference, attribute comparison, set operation, and etc. Existing benchmarks have some shortcomings that limit the development of Complex KBQA: 1) they only provide QA pairs without explicit reasoning processes; 2) questions are either generated by templates, leading to poor diversity, or on a small scale. To this end, we introduce KQA Pro, a large-scale dataset for Complex KBQA. We define a compositional and highly-interpretable formal format, named Program, to represent the reasoning process of complex questions. We propose compositional strategies to generate questions, corresponding SPARQLs, and Programs with a small number of templates, and then paraphrase the generated questions to natural language questions (NLQ) by crowdsourcing, giving rise to around 120K diverse instances. SPARQL and Program depict two complementary solutions to answer complex questions, which can benefit a large spectrum of QA methods. Besides the QA task, KQA Pro can also serves for the semantic parsing task. As far as we know, it is currently the largest corpus of NLQ-to-SPARQL and NLQ-to-Program. We conduct extensive experiments to evaluate whether machines can learn to answer our complex questions in different cases, that is, with only QA supervision or with intermediate SPARQL/Program supervision. We find that state-of-the-art KBQA methods learnt from only QA pairs perform very poor on our dataset, implying our questions are more challenging than previous datasets. However, pretrained models learnt from our NLQ-to-SPARQL and NLQ-to-Program annotations surprisingly achieve about 90\% answering accuracy, which is even close to the human expert performance...Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- SPARQA: Skeleton-Based Semantic Parsing for Complex Questions over Knowledge BasesProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Stepwise Reasoning for Multi-Relation Question Answering over Knowledge Graph with Weak Supervision2020
- Hierarchical Graph Network for Multi-hop Question Answering2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings2020
- Complex Program Induction for Querying Knowledge Bases in the Absence of Gold ProgramsTransactions of the Association for Computational Linguistics, 2019
- Explainable and Explicit Visual Reasoning Over Scene Graphs2019
- Knowledge Graph Embedding Based Question Answering2019
- Variational Reasoning for Question Answering With Knowledge GraphProceedings of the AAAI Conference on Artificial Intelligence, 2018
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning2017