Query Performance Prediction under Corpus Growth in Dense Retrieval
Open Access
- 19 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3733-3737
Abstract
LLM-based chatbots are increasingly augmented with retrieval mechanisms operating over web-scale corpora. Evaluating the effectiveness of these retrieval components is challenging, as explicit relevance judgments are often unavailable. Query performance prediction (QPP) addresses this limitation by providing unsupervised estimates of retrieval effectiveness. However, existing QPP methods assume a static corpus and do not account for the impact of corpus growth on query performance. In this work, we extend the QPP paradigm by studying query performance degradation under corpus inflation in dense retrieval systems. Using tiered corpora with fixed relevance judgments, we analyze how query effectiveness evolves as the corpus (index) size increases and evaluate the ability of established score-based and embedding-based post-retrieval QPP methods to predict such degradation. Our findings show that the reliability of these predictors is dependent on the dataset. We propose simple adaptations to established QPP measures, most notably a top-k vs background Wasserstein distance measure, which yield more consistent associations with degradation and outperform their original counterparts. These findings highlight limitations of several QPP approaches in large-scale, continuously expanding retrieval environments and motivate the development of corpus-growth-aware QPP measures.Keywords
This publication has 16 references indexed in Scilit:
- An Analysis of Variations in the Effectiveness of Query Performance Prediction2022
- sMARE: a new paradigm to evaluate and understand query performance prediction methodsInformation Retrieval Journal, 2022
- Robust Standard Deviation Estimation for Query Performance Prediction2017
- Query Performance Prediction By Considering Score Magnitude and Variance Together2014
- Predicting Query Performance by Query-Drift EstimationACM Transactions on Information Systems, 2012
- A case for improved evaluation of query difficulty prediction2009
- Effective Pre-retrieval Query Performance Prediction Using Similarity and Variability Evidence2008
- Query performance prediction in web search environments2007
- Query performance predictionInformation Systems, 2006
- Query association surrogates for Web searchJournal of the American Society for Information Science and Technology, 2004