Questions Are All You Need to Train a Dense Passage Retriever
Preprint
Abstract
We introduce ART, a new corpus-level autoencoding approach for training dense retrieval models that does not require any labeled training data. Dense retrieval is a central challenge for open-domain tasks, such as Open QA, where state-of-the-art methods typically require large supervised datasets with custom hard-negative mining and denoising of positive examples. ART, in contrast, only requires access to unpaired inputs and outputs (e.g. questions and potential answer documents). It uses a new document-retrieval autoencoding scheme, where (1) an input question is used to retrieve a set of evidence documents, and (2) the documents are then used to compute the probability of reconstructing the original question. Training for retrieval based on question reconstruction enables effective unsupervised learning of both document and question encoders, which can be later incorporated into complete Open QA systems without any further finetuning. Extensive experiments demonstrate that ART obtains state-of-the-art results on multiple QA retrieval benchmarks with only generic initialization from a pre-trained language model, removing the need for labeled data and task-specific losses.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Competition-level code generation with AlphaCodeScience, 2022
- InPars: Unsupervised Dataset Generation for Information Retrieval2022
- Relevance-guided Supervision for OpenQA with ColBERTTransactions of the Association for Computational Linguistics, 2021
- Sparse, Dense, and Attentional Representations for Text RetrievalTransactions of the Association for Computational Linguistics, 2021
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse LanguagesTransactions of the Association for Computational Linguistics, 2020
- ColBERT2020
- Document Ranking with a Pretrained Sequence-to-Sequence Model2020
- Beyond [CLS] through Ranking by Generation2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- The Probabilistic Relevance Framework: BM25 and BeyondFoundations and Trends® in Information Retrieval, 2009