TriSampler: A Better Negative Sampling Principle for Dense Retrieval
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (8) , 9269-9277
Abstract
Negative sampling stands as a pivotal technique in dense retrieval, essential for training effective retrieval models and significantly impacting retrieval performance. While existing negative sampling methods have made commendable progress by leveraging hard negatives, a comprehensive guiding principle for constructing negative candidates and designing negative sampling distributions is still lacking. To bridge this gap, we embark on a theoretical analysis of negative sampling in dense retrieval. This exploration culminates in the unveiling of the quasi-triangular principle, a novel framework that elucidates the triangular-like interplay between query, positive document, and negative document. Fueled by this guiding principle, we introduce TriSampler, a straightforward yet highly effective negative sampling method. The keypoint of TriSampler lies in its ability to selectively sample more informative negatives within a prescribed constrained region. Experimental evaluation show that TriSampler consistently attains superior retrieval performance across a diverse of representative retrieval models.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval2022
- Sparse, Dense, and Attentional Representations for Text RetrievalTransactions of the Association for Computational Linguistics, 2021
- TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning2021
- RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking2021
- End-to-End Training of Neural Retrievers for Open-Domain Question Answering2021
- PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval2021
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering2021
- On the Sentence Embeddings from Pre-trained Language Models2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019