Anna Noel-Storr
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Anna Noel-Storr has a total of 451 co-authors. Here are the top co-authors by number of shared publications:
James Thomas
EPPI Centre, UCL Social Research Institute, University College London, London, UK
4 shared publicationsIan Shemilt
EPPI Centre, UCL Social Research Institute, University College London, London, UK
3 shared publications- James Thomas
EPPI-Centre, UCL Social Research Institute, University College London, Gower St, London, WC1E 6BT, UK
3 shared publications - A. Joy Allen
NIHR Newcastle In Vitro Diagnostics Co-operative, Room M2.088, Translational and Clinical Research Institute, William Leech Building, Medical School, Newcastle University, Newcastle NE2 4HH, UK
2 shared publications Adele Francis
52Department of Breast Surgery, Nuffield House, University Hospital Birmingham, Birmingham, UK
2 shared publications- Adina Najwa Kamarudin
112Department of Biostatistics, University of Liverpool, Liverpool, UK
2 shared publications
Latest Publications
- Open Access
INSPECT-SR tool for assessing trustworthiness of randomised controlled trials
The integrity of evidence synthesis is threatened by problematic randomised controlled trials, where there are serious concerns about the trustworthiness of the data or findings. Such concerns could be due to research misconduct, including fraud, or due to honest critical errors. If these trials are not detected, they may be inadvertently included in systematic reviews and guidelines, potentially distorting their results. To resolve this problem, the INSPECT-SR (Investigating Problematic Clinical Trials in Systematic Reviews) tool has been developed to assess the trustworthiness of randomised controlled trials. This tool will help researchers to identify and exclude problematic trials from systematic reviews. This article describes the development of INSPECT-SR and presents the tool and an associated guidance document.
Open AccessINSPECT-SR tool for assessing trustworthiness of randomised controlled trials
The integrity of evidence synthesis is threatened by problematic randomised controlled trials, where there are serious concerns about the trustworthiness of the data or findings. Such concerns could be due to research misconduct, including fraud, or due to honest critical errors. If these trials are not detected, they may be inadvertently included in systematic reviews and guidelines, potentially distorting their results. To resolve this problem, the INSPECT-SR (Investigating Problematic Clinical Trials in Systematic Reviews) tool has been developed to assess the trustworthiness of randomised controlled trials. This tool will help researchers to identify and exclude problematic trials from systematic reviews. This article describes the development of INSPECT-SR and presents the tool and an associated guidance document.
- Open Access
HPV Self-Sampling in Cervical Screening: A Rapid Review
Introduction: Cervical cancer is the fourth largest cause of cancer deaths in women. HPV self-sampling could increase uptake of cervical screening. This rapid review aimed to determine the accuracy, concordance, uptake and acceptability of self-sampling over clinician-collected samples in high income countries. Method: We followed Cochrane Rapid Reviews Methods. Top-up of 4 systematic reviews and meta-analyses was performed. Narrative data synthesis was conducted and meta-analysis where applicable. Databases searched were MEDLINE, EMBASE, CENTRAL and clinical trial registries. Risk of bias was assessed using AMSTAR 2, QUADAS, the Cochrane Risk of Bias (RoB), or the Nudelman and Otto, 2020 tool, depending on the study type. Findings: The review included 39 studies for accuracy, 38 studies for concordance, 37 uptake and 48 studies for acceptability. Self-sampling has similar accuracy as clinician-collected samples when PCR-based assays are used. The overall agreement of self-sampling and clinician-collected samples was 87.1%(95%CI;85.6-88.6) with a kappa value of 0.70(95%CI;0.67-0.73). Mail-to-all strategies had higher uptake with participation differences of 11.3%(95%CI:8.4-14.2) in the intention-to-treat analysis and 7.7%(95%CI:4.7-10.8) in the per protocol analysis. Self-sampling is acceptable to non-attendees (91%(95%CI;85.3-94.6). Conclusion and Recommendation: Self-sampling shows good performance on the four clinical effectiveness indicators of accuracy, concordance, uptake and acceptability.
Open AccessHPV Self-Sampling in Cervical Screening: A Rapid Review
Introduction: Cervical cancer is the fourth largest cause of cancer deaths in women. HPV self-sampling could increase uptake of cervical screening. This rapid review aimed to determine the accuracy, concordance, uptake and acceptability of self-sampling over clinician-collected samples in high income countries. Method: We followed Cochrane Rapid Reviews Methods. Top-up of 4 systematic reviews and meta-analyses was performed. Narrative data synthesis was conducted and meta-analysis where applicable. Databases searched were MEDLINE, EMBASE, CENTRAL and clinical trial registries. Risk of bias was assessed using AMSTAR 2, QUADAS, the Cochrane Risk of Bias (RoB), or the Nudelman and Otto, 2020 tool, depending on the study type. Findings: The review included 39 studies for accuracy, 38 studies for concordance, 37 uptake and 48 studies for acceptability. Self-sampling has similar accuracy as clinician-collected samples when PCR-based assays are used. The overall agreement of self-sampling and clinician-collected samples was 87.1%(95%CI;85.6-88.6) with a kappa value of 0.70(95%CI;0.67-0.73). Mail-to-all strategies had higher uptake with participation differences of 11.3%(95%CI:8.4-14.2) in the intention-to-treat analysis and 7.7%(95%CI:4.7-10.8) in the per protocol analysis. Self-sampling is acceptable to non-attendees (91%(95%CI;85.3-94.6). Conclusion and Recommendation: Self-sampling shows good performance on the four clinical effectiveness indicators of accuracy, concordance, uptake and acceptability.
- Open Access
- 1 December 2025
- Published by National Institute for Health and Care Research in Health Technology Assessment
- p. 1-38
Risk assessment tools for predicting transfusion in surgery: a systematic review and meta-analysis
Background: United Kingdom blood shortages necessitate better prediction of surgical blood requirement. We sought to assess the predictive accuracy of tools designed to identify those patients requiring blood transfusion within the perioperative period. Methods: We searched the Cochrane library, EMBASE, MEDLINE, ClinicalTrials.gov and WHO trials portal, 2000–July 2023. We included studies that developed and/or validated prediction tools for blood requirement during the early perioperative period (48 hours). Risk of bias was evaluated using the Prediction model Risk Of Bias Assessment Tool. We pooled area under receiver operating curve and calibration data via random effects meta-analysis. We evaluated certainty of evidence of any estimates using the Grading of Recommendations Assessment, Development and Evaluation framework. We used meta-regression to describe associations between included variables/tool characteristics with tool accuracy. Results: We included 50 papers, describing 67 unique prediction tools. Most tools were at high risk of bias, with limited external validation. Discrimination (area under receiver operating curve) of prognostic models ranged from 0.49 to 0.96. Only two surgery-specific tools, the McClusky Index (liver transplant surgery) and Papworth Bleeding Risk Score (cardiothoracic surgery), had sufficient data to enable pooling of discrimination measures. The McClusky Index’s pooled area under receiver operating curve: 0.74 (95% CI 0.61 to 0.84) and Bleeding Risk Score’s area under receiver operating curve: 0.68 (95% CI 0.49 to 0.82) were both rated ‘very low’ certainty by Grading of Recommendations Assessment, Development and Evaluation. Pooling calibration data was not possible for any prediction tools. Meta-regression suggested that fewer included variables, longer time from surgery and independent validation studies were all associated with lower accuracy. Limitations: There were insufficient studies to assess overall tool performance via meta-analysis in other surgical subgroups beyond cardiothoracic surgery and liver transplant. Our study population is also predominantly made up of elective surgeries which may make our results less generalisable to emergency settings. Future work: Implementation and cost-effectiveness studies are needed to evaluate how promising tools could be applied to clinical practice and the economic impact such tools could have upon the service. Conclusions: Despite the availability of multiple potential tools, available data suggest none are currently suitable for predicting blood transfusion in surgical practice. Our summary of the data comes with caveats around the quality of the included papers and the limited number of tools with more than one reported external validation. Funding: This article presents independent research funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme as award number NIHR159933.
- 1 December 2025
- Published by National Institute for Health and Care Research in Health Technology Assessment
- p. 1-38
Open AccessRisk assessment tools for predicting transfusion in surgery: a systematic review and meta-analysis
Background: United Kingdom blood shortages necessitate better prediction of surgical blood requirement. We sought to assess the predictive accuracy of tools designed to identify those patients requiring blood transfusion within the perioperative period. Methods: We searched the Cochrane library, EMBASE, MEDLINE, ClinicalTrials.gov and WHO trials portal, 2000–July 2023. We included studies that developed and/or validated prediction tools for blood requirement during the early perioperative period (48 hours). Risk of bias was evaluated using the Prediction model Risk Of Bias Assessment Tool. We pooled area under receiver operating curve and calibration data via random effects meta-analysis. We evaluated certainty of evidence of any estimates using the Grading of Recommendations Assessment, Development and Evaluation framework. We used meta-regression to describe associations between included variables/tool characteristics with tool accuracy. Results: We included 50 papers, describing 67 unique prediction tools. Most tools were at high risk of bias, with limited external validation. Discrimination (area under receiver operating curve) of prognostic models ranged from 0.49 to 0.96. Only two surgery-specific tools, the McClusky Index (liver transplant surgery) and Papworth Bleeding Risk Score (cardiothoracic surgery), had sufficient data to enable pooling of discrimination measures. The McClusky Index’s pooled area under receiver operating curve: 0.74 (95% CI 0.61 to 0.84) and Bleeding Risk Score’s area under receiver operating curve: 0.68 (95% CI 0.49 to 0.82) were both rated ‘very low’ certainty by Grading of Recommendations Assessment, Development and Evaluation. Pooling calibration data was not possible for any prediction tools. Meta-regression suggested that fewer included variables, longer time from surgery and independent validation studies were all associated with lower accuracy. Limitations: There were insufficient studies to assess overall tool performance via meta-analysis in other surgical subgroups beyond cardiothoracic surgery and liver transplant. Our study population is also predominantly made up of elective surgeries which may make our results less generalisable to emergency settings. Future work: Implementation and cost-effectiveness studies are needed to evaluate how promising tools could be applied to clinical practice and the economic impact such tools could have upon the service. Conclusions: Despite the availability of multiple potential tools, available data suggest none are currently suitable for predicting blood transfusion in surgical practice. Our summary of the data comes with caveats around the quality of the included papers and the limited number of tools with more than one reported external validation. Funding: This article presents independent research funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme as award number NIHR159933.
- Open Access
- 25 April 2022
- Published by Wiley in Research Synthesis Methods
- Vol. 13 (5) , 585-594
Crowdsourcing the identification of studies for COVID‐19‐related Cochrane Rapid Reviews
Background: Utilisation of crowdsourcing within evidence synthesis has increased over the last decade. Crowdsourcing platform Cochrane Crowd has engaged a global community of 22,000 people from 170 countries. The COVID‐19 pandemic presented an opportunity to engage the community and keep up with the exponential output of COVID‐19 research.Aims: To test whether a crowd could accurately assess study eligibility for reviews under time constraints. Outcome measures: time taken to complete each task, time to produce required training modules, crowd sensitivity, specificity and crowd consensus.Methods: We created four crowd tasks, corresponding to four Cochrane COVID‐19 Rapid Reviews. The search results of each were uploaded and an interactive training module was developed for each task. Contributors who had participated in another COVID‐19 task were invited to participate. Each task was live for 48‐h. The final inclusion and exclusion decisions made by the core author team were used as the reference standard.Results: Across all four reviews 14,299 records were screened by 101 crowd contributors. The crowd completed each screening task within 48‐h for three reviews and in 52 h for one. Sensitivity ranged from 94% to 100%. Four studies, out of a total of 109, were incorrectly rejected by the crowd. However, their absence ultimately would not have altered the conclusions of the reviews. Crowd consensus ranged from 71% to 92% across the four reviews.Conclusion: Crowdsourcing can play a valuable role in study identification and offers willing contributors the opportunity to help identify COVID‐19 research for rapid evidence syntheses.
- 25 April 2022
- Published by Wiley in Research Synthesis Methods
- Vol. 13 (5) , 585-594
Open AccessCrowdsourcing the identification of studies for COVID‐19‐related Cochrane Rapid Reviews
Background: Utilisation of crowdsourcing within evidence synthesis has increased over the last decade. Crowdsourcing platform Cochrane Crowd has engaged a global community of 22,000 people from 170 countries. The COVID‐19 pandemic presented an opportunity to engage the community and keep up with the exponential output of COVID‐19 research.Aims: To test whether a crowd could accurately assess study eligibility for reviews under time constraints. Outcome measures: time taken to complete each task, time to produce required training modules, crowd sensitivity, specificity and crowd consensus.Methods: We created four crowd tasks, corresponding to four Cochrane COVID‐19 Rapid Reviews. The search results of each were uploaded and an interactive training module was developed for each task. Contributors who had participated in another COVID‐19 task were invited to participate. Each task was live for 48‐h. The final inclusion and exclusion decisions made by the core author team were used as the reference standard.Results: Across all four reviews 14,299 records were screened by 101 crowd contributors. The crowd completed each screening task within 48‐h for three reviews and in 52 h for one. Sensitivity ranged from 94% to 100%. Four studies, out of a total of 109, were incorrectly rejected by the crowd. However, their absence ultimately would not have altered the conclusions of the reviews. Crowd consensus ranged from 71% to 92% across the four reviews.Conclusion: Crowdsourcing can play a valuable role in study identification and offers willing contributors the opportunity to help identify COVID‐19 research for rapid evidence syntheses.
- Open Access
- 22 January 2022
- Published by Springer Nature in Systematic Reviews
- Vol. 11 (1) , 1-8
Machine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier
Background: This study developed, calibrated and evaluated a machine learning (ML) classifier designed to reduce study identification workload in maintaining the Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies. Methods: A ML classifier for retrieving COVID-19 research studies (the ‘Cochrane COVID-19 Study Classifier’) was developed using a data set of title-abstract records ‘included’ in, or ‘excluded’ from, the CCSR up to 18th October 2020, manually labelled by information and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between October 19 and December 2, 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between the 4th and 19th of January 2021. Results: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were ‘included’ in the CCSR). A classification threshold was set using 16,123 calibration records (6005 of which were ‘included’ in the CCSR) and the classifier had a precision of 0.52 in this data set at the target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2285 (98.9%) of 2310 eligible records but missed 25 (1%), with a precision of 0.638 and a net screening workload reduction of 24.1% (1113 records correctly excluded). Conclusions: The Cochrane COVID-19 Study Classifier reduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.
- 22 January 2022
- Published by Springer Nature in Systematic Reviews
- Vol. 11 (1) , 1-8
Open AccessMachine learning reduced workload for the Cochrane COVID-19 Study Register: development and evaluation of the Cochrane COVID-19 Study Classifier
Background: This study developed, calibrated and evaluated a machine learning (ML) classifier designed to reduce study identification workload in maintaining the Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies. Methods: A ML classifier for retrieving COVID-19 research studies (the ‘Cochrane COVID-19 Study Classifier’) was developed using a data set of title-abstract records ‘included’ in, or ‘excluded’ from, the CCSR up to 18th October 2020, manually labelled by information and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between October 19 and December 2, 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between the 4th and 19th of January 2021. Results: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were ‘included’ in the CCSR). A classification threshold was set using 16,123 calibration records (6005 of which were ‘included’ in the CCSR) and the classifier had a precision of 0.52 in this data set at the target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2285 (98.9%) of 2310 eligible records but missed 25 (1%), with a precision of 0.638 and a net screening workload reduction of 24.1% (1113 records correctly excluded). Conclusions: The Cochrane COVID-19 Study Classifier reduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.
- Open Access
- 14 July 2021
- Published by Springer Nature in Research Square
Machine Learning Reduced Workload for the Cochrane COVID-19 Study Register: Development and Evaluation of the Cochrane COVID-19 Study Classifier
Background: This study developed, calibrated, and evaluated a machine learning (ML) classifier designed to reduce study identification workload inmaintainingthe Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies.Methods: A ML classifier for retrieving COVID-19 research studies (the “Cochrane COVID-19 Study Classifier”) was developedusing a data set of title-abstract records ‘included’ in, or‘excluded’ from,the CCSR up to18th October 2020, manually labelled byinformation and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records ‘included’ in, or ‘excluded’ from,theCCSRbetween 19th October and 2ndDecember 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between4thand 19thJanuary 2021.Results: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were ‘included’ in the CCSR). A classification threshold was set using 16,123 calibration records (6,005 of which were ‘included’ in the CCSR) and the classifier hada precision of 0.52in this dataset atthe target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2,285 (98.9%) of 2,310eligible study reports but missed 25 (1%), with aprecision of 0.638 and a netscreening workload reduction of 24.1% (1,113 records correctly excluded). Conclusions: The Cochrane COVID-19 Study Classifierreduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.
- 14 July 2021
- Published by Springer Nature in Research Square
Open AccessMachine Learning Reduced Workload for the Cochrane COVID-19 Study Register: Development and Evaluation of the Cochrane COVID-19 Study Classifier
Background: This study developed, calibrated, and evaluated a machine learning (ML) classifier designed to reduce study identification workload inmaintainingthe Cochrane COVID-19 Study Register (CCSR), a continuously updated register of COVID-19 research studies.Methods: A ML classifier for retrieving COVID-19 research studies (the “Cochrane COVID-19 Study Classifier”) was developedusing a data set of title-abstract records ‘included’ in, or‘excluded’ from,the CCSR up to18th October 2020, manually labelled byinformation and data curation specialists or the Cochrane Crowd. The classifier was then calibrated using a second data set of similar records ‘included’ in, or ‘excluded’ from,theCCSRbetween 19th October and 2ndDecember 2020, aiming for 99% recall. Finally, the calibrated classifier was evaluated using a third data set of similar records ‘included’ in, or ‘excluded’ from, the CCSR between4thand 19thJanuary 2021.Results: The Cochrane COVID-19 Study Classifier was trained using 59,513 records (20,878 of which were ‘included’ in the CCSR). A classification threshold was set using 16,123 calibration records (6,005 of which were ‘included’ in the CCSR) and the classifier hada precision of 0.52in this dataset atthe target threshold recall >0.99. The final, calibrated COVID-19 classifier correctly retrieved 2,285 (98.9%) of 2,310eligible study reports but missed 25 (1%), with aprecision of 0.638 and a netscreening workload reduction of 24.1% (1,113 records correctly excluded). Conclusions: The Cochrane COVID-19 Study Classifierreduces manual screening workload for identifying COVID-19 research studies, with a very low and acceptable risk of missing eligible studies. It is now deployed in the live study identification workflow for the Cochrane COVID-19 Study Register.
- Top Cited PapersOpen Access
- 1 May 2021
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 133, 130-139
An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials
- 1 May 2021
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 133, 130-139
Top Cited PapersOpen AccessAn evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials
- Open Access
- 26 April 2021
- Published by Springer Nature in BMC Medical Research Methodology
- Vol. 21 (1) , 1-10
Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study
Background: Crowdsourcing engages the help of large numbers of people in tasks, activities or projects, usually via the internet. One application of crowdsourcing is the screening of citations for inclusion in a systematic review. There is evidence that a ‘Crowd’ of non-specialists can reliably identify quantitative studies, such as randomized controlled trials, through the assessment of study titles and abstracts. In this feasibility study, we investigated crowd performance of an online, topic-based citation-screening task, assessing titles and abstracts for inclusion in a single mixed-studies systematic review. Methods: This study was embedded within a mixed studies systematic review of maternity care, exploring the effects of training healthcare professionals in intrapartum cardiotocography. Citation-screening was undertaken via Cochrane Crowd, an online citizen science platform enabling volunteers to contribute to a range of tasks identifying evidence in health and healthcare. Contributors were recruited from users registered with Cochrane Crowd. Following completion of task-specific online training, the crowd and the review team independently screened 9546 titles and abstracts. The screening task was subsequently repeated with a new crowd following minor changes to the crowd agreement algorithm based on findings from the first screening task. We assessed the crowd decisions against the review team categorizations (the ‘gold standard’), measuring sensitivity, specificity, time and task engagement. Results: Seventy-eight crowd contributors completed the first screening task. Sensitivity (the crowd’s ability to correctly identify studies included within the review) was 84% (N = 42/50), and specificity (the crowd’s ability to correctly identify excluded studies) was 99% (N = 9373/9493). Task completion was 33 h for the crowd and 410 h for the review team; mean time to classify each record was 6.06 s for each crowd participant and 3.96 s for review team members. Replicating this task with 85 new contributors and an altered agreement algorithm found 94% sensitivity (N = 48/50) and 98% specificity (N = 9348/9493). Contributors reported positive experiences of the task. Conclusion: It might be feasible to recruit and train a crowd to accurately perform topic-based citation-screening for mixed studies systematic reviews, though resource expended on the necessary customised training required should be factored in. In the face of long review production times, crowd screening may enable a more time-efficient conduct of reviews, with minimal reduction of citation-screening accuracy, but further research is needed.
- 26 April 2021
- Published by Springer Nature in BMC Medical Research Methodology
- Vol. 21 (1) , 1-10
Open AccessCrowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study
Background: Crowdsourcing engages the help of large numbers of people in tasks, activities or projects, usually via the internet. One application of crowdsourcing is the screening of citations for inclusion in a systematic review. There is evidence that a ‘Crowd’ of non-specialists can reliably identify quantitative studies, such as randomized controlled trials, through the assessment of study titles and abstracts. In this feasibility study, we investigated crowd performance of an online, topic-based citation-screening task, assessing titles and abstracts for inclusion in a single mixed-studies systematic review. Methods: This study was embedded within a mixed studies systematic review of maternity care, exploring the effects of training healthcare professionals in intrapartum cardiotocography. Citation-screening was undertaken via Cochrane Crowd, an online citizen science platform enabling volunteers to contribute to a range of tasks identifying evidence in health and healthcare. Contributors were recruited from users registered with Cochrane Crowd. Following completion of task-specific online training, the crowd and the review team independently screened 9546 titles and abstracts. The screening task was subsequently repeated with a new crowd following minor changes to the crowd agreement algorithm based on findings from the first screening task. We assessed the crowd decisions against the review team categorizations (the ‘gold standard’), measuring sensitivity, specificity, time and task engagement. Results: Seventy-eight crowd contributors completed the first screening task. Sensitivity (the crowd’s ability to correctly identify studies included within the review) was 84% (N = 42/50), and specificity (the crowd’s ability to correctly identify excluded studies) was 99% (N = 9373/9493). Task completion was 33 h for the crowd and 410 h for the review team; mean time to classify each record was 6.06 s for each crowd participant and 3.96 s for review team members. Replicating this task with 85 new contributors and an altered agreement algorithm found 94% sensitivity (N = 48/50) and 98% specificity (N = 9348/9493). Contributors reported positive experiences of the task. Conclusion: It might be feasible to recruit and train a crowd to accurately perform topic-based citation-screening for mixed studies systematic reviews, though resource expended on the necessary customised training required should be factored in. In the face of long review production times, crowd screening may enable a more time-efficient conduct of reviews, with minimal reduction of citation-screening accuracy, but further research is needed.
- Open Access
- 1 February 2021
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 130, 23-31
Citation screening using crowdsourcing and machine learning produced accurate results: Evaluation of Cochrane's modified Screen4Me service
- 1 February 2021
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 130, 23-31
Open AccessCitation screening using crowdsourcing and machine learning produced accurate results: Evaluation of Cochrane's modified Screen4Me service
- Open Access
- 1 November 2020
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 127, 142-150
Cochrane Centralised Search Service showed high sensitivity identifying randomized controlled trials: A retrospective analysis
- 1 November 2020
- Published by Elsevier in Journal of Clinical Epidemiology
- Vol. 127, 142-150
Open AccessCochrane Centralised Search Service showed high sensitivity identifying randomized controlled trials: A retrospective analysis