A Test Collection for Evaluating Retrieval of Studies for Inclusion in Systematic Reviews
Open Access
- 7 August 2017
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1237-1240
Abstract
This paper introduces a test collection for evaluating the effectiveness of different methods used to retrieve research studies for inclusion in systematic reviews. Systematic reviews appraise and synthesise studies that meet specific inclusion criteria. Systematic reviews intended for a biomedical science audience use boolean queries with many, often complex, search clauses to retrieve studies; these are then manually screened to determine eligibility for inclusion in the review. This process is expensive and time consuming. The development of systems that improve retrieval effectiveness will have an immediate impact by reducing the complexity and resources required for this process. Our test collection consists of approximately 26 million research studies extracted from the freely available MEDLINE database, 94 review (query) topics extracted from Cochrane systematic reviews, and corresponding relevance assessments. Tasks for which the collection can be used for information retrieval system evaluation are described and the use of the collection to evaluate common baselines within one such task is demonstrated. The test collection is available at https://github.com/ielab/SIGIR2017-PICO-CollectionKeywords
This publication has 11 references indexed in Scilit:
- Searching and Stopping2015
- Erratum to: Using text mining for study identification in systematic reviews: a systematic review of current approachesSystematic Reviews, 2015
- Using text mining for study identification in systematic reviews: a systematic review of current approachesSystematic Reviews, 2015
- Systematic review automation technologiesSystematic Reviews, 2014
- Feature Engineering and a Proposed Decision-Support System for Systematic Reviewers of Medical EvidencePLOS ONE, 2014
- Pinpointing needles in giant haystacks: use of text mining to reduce impractical screening workload in extremely large scoping reviewsResearch Synthesis Methods, 2013
- Performance of support-vector-machine-based classification on 15 systematic review topics evaluated with the WSS@95 measure: Table 1Journal of the American Medical Informatics Association, 2011
- Boolean versus ranked querying for biomedical systematic reviewsBMC Medical Informatics and Decision Making, 2010
- Utilization of the PICO framework to improve searching PubMed for clinical questionsBMC Medical Informatics and Decision Making, 2007
- Reducing Workload in Systematic Review Preparation Using Automated Citation ClassificationJournal of the American Medical Informatics Association, 2006