Topic set size redux
Author:
Open Access
- 19 July 2009
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 806-807
Abstract
The cost as well as the power and reliability of a retrieval test collection are all proportional to the number of topics included in it. Test collections created through community evaluations such as TREC generally use 50 topics. Prior work estimated the reliability of 50-topic sets by extrapolating confidence levels from those of smaller sets, and concluded that 50 topics are sufficient to have high confidence in a comparison, especially when the comparison is statistically significant. Using topic sets that actually contain 50 topics, this paper shows that statistically significant differences can be wrong, even when statistical significance is accompanied by moderately large (>10%) relative differences in scores. Further, using standardized evaluation scores rather than raw evaluation scores does not increase the reliability of these paired comparisons. Researchers should continue to be skeptical of conclusions demonstrated on only a single test collection.Keywords
This publication has 7 references indexed in Scilit:
- Evaluation over thousands of queries2008
- Score standardization for inter-collection comparison of retrieval systems2008
- Evaluating evaluation metrics based on the bootstrap2006
- Information retrieval system evaluation2005
- The effect of topic set size on retrieval experiment error2002
- Evaluating evaluation measure stability2000
- Blind Men and Elephants: Six Approaches to TREC dataInformation Retrieval Journal, 1999