SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval
Open Access
- 18 July 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2964-2974
Abstract
Traditionally, sparse retrieval systems relied on lexical representations to retrieve documents, such as BM25, dominated information retrieval tasks. With the onset of pre-trained transformer models such as BERT, neural sparse retrieval has led to a new paradigm within retrieval. Despite the success, there has been limited software supporting different sparse retrievers running in a unified, common environment. This hinders practitioners from fairly comparing different sparse models and obtaining realistic evaluation results. Another missing piece is, that a majority of prior work evaluates sparse retrieval models on in-domain retrieval, i.e. on a single dataset: MS MARCO. However, a key requirement in practical retrieval systems requires models that can generalize well to unseen out-of-domain, i.e. zero-shot retrieval tasks. In this work, we provide SPRINT, a unified python toolkit based on Pyserini and Lucene, supporting a common interface for evaluating neural sparse retrieval. The toolkit currently includes five built-in models: uniCOIL, DeepImpact, SPARTA, TILDEv2 and SPLADEv2. Users can also easily add customized models by defining their term weighting method. Using our toolkit, we establish strong and reproducible zero-shot sparse retrieval baselines across the well-acknowledged benchmark, BEIR. Our results demonstrate that SPLADEv2 achieves the best average score of 0.470 [email protected] on BEIR amongst all neural sparse retrievers. In this work, we further uncover the reasons behind its performance gain. We show that SPLADEv2 produces sparse representations with a majority of tokens outside of the original query and document which is often crucial for its performance gains, i.e. a limitation among its other sparse counterparts. We provide our SPRINT toolkit, models, and data used in our experiments publicly here: https://github.com/thakur-nandan/sprint.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking2021
- Pretrained Transformers for Text Ranking: BERT and Beyond2021
- Flexible IR Pipelines with Capreolus2020
- Declarative Experimentation in Information Retrieval using PyTerrier2020
- Context-Aware Term Weighting For First Stage Passage Retrieval2020
- Dense Passage Retrieval for Open-Domain Question Answering2020
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- From Neural Re-Ranking to Neural Ranking2018
- The Probabilistic Relevance Framework: BM25 and BeyondFoundations and Trends® in Information Retrieval, 2009
- Bridging the lexical chasm2000