ANTIQUE: A Non-factoid Question Answering Benchmark
- 8 April 2020
- book chapter
- Published by Springer Nature
- Vol. 12036, 166-173
Abstract
Considering the widespread use of mobile and voice search, answer passage retrieval for non-factoid questions plays a critical role in modern information retrieval systems. Despite the importance of the task, the community still feels the significant lack of large-scale non-factoid question answering collections with real questions and comprehensive relevance judgments. In this paper, we develop and release a collection of 2,626 open-domain non-factoid questions from a diverse set of categories. The dataset, called ANTIQUE, contains 34k manual relevance annotations. The questions were asked by real users in a community question answering service, i.e., Yahoo! Answers. Relevance judgments for all the answers to each question were collected through crowdsourcing. To facilitate further research, we also include a brief analysis of the data as well as baseline results on both classical and neural IR models.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- WikiPassageQA2018
- A Hybrid Embedding Approach to Noisy Answer Passage Retrieval2018
- SemEval-2017 Task 3: Community Question Answering2017
- A Deep Relevance Matching Model for Ad-hoc Retrieval2016
- aNMM2016
- End to End Long Short Term Memory Networks for Non-Factoid Question Answering2016
- New Collection Announcement2016
- Applying deep learning to answer selection: A study and an open task2015
- Evaluating answer passages using summarization measures2014
- Evaluating and predicting answer quality in community QA2010