The Best Published Result is Random
Open Access
- 9 August 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 747-750
Abstract
Reusable test collections allow researchers to rapidly test different algorithms to find the one that works "best". But because of randomness in the topic sample, or in relevance judgments, or in interactions among system components, extreme results can be seen entirely due to chance, particularly when a collection becomes very popular. We argue that the best known published effectiveness on any given collection could be measured as much as 20% higher than its "true" intrinsic effectiveness, and that there are many other systems with lower measured effectiveness that could have substantially higher intrinsic effectiveness.Keywords
Funding Information
- NSF (IIS-1350799)
This publication has 4 references indexed in Scilit:
- Multiple testing in statistical analysis of systems-based information retrieval experimentsACM Transactions on Information Systems, 2012
- A comparison of statistical significance tests for information retrieval evaluation2007
- An Introduction to Statistical Modeling of Extreme Values2001
- The seventh text REtrieval conference (TREC-7)1999