A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks
Authors:
- Nicholas MonathNMNicholas Monath
- Will GrathwohlWGWill Grathwohl
- Michael BoratkoMBMichael Boratko
- Rob FergusRFRob Fergus
- Andrew McCallumAMAndrew McCallum
- Manzil ZaheerMZManzil Zaheer
Preprint
Abstract
In dense retrieval, deep encoders provide embeddings for both inputs and targets, and the softmax function is used to parameterize a distribution over a large number of candidate targets (e.g., textual passages for information retrieval). Significant challenges arise in training such encoders in the increasingly prevalent scenario of (1) a large number of targets, (2) a computationally expensive target encoder model, (3) cached target embeddings that are out-of-date due to ongoing training of target encoder parameters. This paper presents a simple and highly scalable response to these challenges by training a small parametric corrector network that adjusts stale cached target embeddings, enabling an accurate softmax approximation and thereby sampling of up-to-date high scoring "hard negatives." We theoretically investigate the generalization properties of our proposed target corrector, relating the complexity of the network, staleness of cached representations, and the amount of training data. We present experimental results on large benchmark dense retrieval datasets as well as on QA with retrieval augmented language models. Our approach matches state-of-the-art results even when no target embedding updates are made during training beyond an initial cache from the unsupervised pre-trained model, providing a 4-80x reduction in re-embedding computational cost.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Large Dual Encoders Are Generalizable Retrievers2022
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering2021
- How Much Knowledge Can You Pack Into the Parameters of a Language Model?2020
- Dense Passage Retrieval for Open-Domain Question Answering2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Learning Dense Representations for Entity Retrieval2019
- Zero-Shot Entity Linking by Reading Entity Descriptions2019
- A Probabilistic Theory of Pattern Recognition1996