Combining Query Performance Predictors: A Reproducibility Study
Preprint
Abstract
A large number of approaches to Query Performance Prediction (QPP) have been proposed over the last two decades. As early as 2009, Hauff et al. [28] explored whether different QPP methods may be combined to improve prediction quality. Since then, significant research has been done both on QPP approaches, as well as their evaluation. This study revisits Hauff et al.s work to assess the reproducibility of their findings in the light of new prediction methods, evaluation metrics, and datasets. We expand the scope of the earlier investigation by: (i) considering post-retrieval methods, including supervised neural techniques (only pre-retrieval techniques were studied in [28]); (ii) using sMARE for evaluation, in addition to the traditional correlation coefficients and RMSE; and (iii) experimenting with additional datasets (Clueweb09B and TREC DL). Our results largely support previous claims, but we also present several interesting findings. We interpret these findings by taking a more nuanced look at the correlation between QPP methods, examining whether they capture diverse information or rely on overlapping factors.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- An in-depth investigation on the behavior of measures to quantify reproducibilityInformation Processing & Management, 2023
- Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance predictionInformation Processing & Management, 2019
- Query Performance Prediction Using Reference ListsACM Transactions on Information Systems, 2016
- Predicting Query Performance by Query-Drift EstimationACM Transactions on Information Systems, 2012
- Efficient and effective spam filtering and re-ranking for large web datasetsInformation Retrieval Journal, 2011
- Predicting the effectiveness of queries and retrieval systemsACM SIGIR Forum, 2010
- Bolasso2008
- Precision prediction based on ranked list coherenceInformation Retrieval Journal, 2006
- Introduction to WordNet: An On-line Lexical Database*International Journal of Lexicography, 1990
- Development and application of a metric on semantic netsIEEE Transactions on Systems, Man, and Cybernetics, 1989