Using an artificial intelligence tool can be as accurate as human assessors in level one screening for a systematic review
- 18 November 2021
- journal article
- review article
- Published by Wiley in Health Information & Libraries Journal
- Vol. 41 (2) , 136-148
Abstract
Background: Artificial intelligence (AI) offers a promising solution to expedite various phases of the systematic review process such as screening. Objective: We aimed to assess the accuracy of an AI tool in identifying eligible references for a systematic review compared to identification by human assessors. Methods: For the case study (a systematic review of knowledge translation interventions), we used a diagnostic accuracy design and independently assessed for eligibility a set of articles ( n = 300) using human raters and the AI system DistillerAI (Evidence Partners, Ottawa, Canada). We analysed a series of 64 possible confidence levels for the AI’s decisions and calculated several standard parameters of diagnostic accuracy for each. Results: When set to a lower AI confidence threshold of 0.1 or greater and an upper threshold of 0.9 or lower, DistillerAI made article selection decisions very similarly to human assessors. Within this range, DistillerAI made a decision on the majority of articles (93–100%), with a sensitivity of 1.0 and specificity ranging from 0.9 to 1.0. Conclusion: DistillerAI appears to be accurate in its assessment of articles in a case study of 300 articles. Further experimentation with DistillerAI will establish its performance among other subject areas.Keywords
This publication has 11 references indexed in Scilit:
- Assessing the accuracy of machine-assisted abstract screening with DistillerAI: a user studySystematic Reviews, 2019
- Prioritising references for systematic reviews with RobotAnalyst: A user studyResearch Synthesis Methods, 2018
- Sensitivity, Specificity, and Predictive Values: Foundations, Pliabilities, and Pitfalls in Research and PracticeFrontiers in Public Health, 2017
- STARD 2015: an updated list of essential items for reporting diagnostic accuracy studiesBMJ, 2015
- RobotReviewer: evaluation of a system for automatically assessing bias in clinical trialsJournal of the American Medical Informatics Association, 2015
- Feature Engineering and a Proposed Decision-Support System for Systematic Reviewers of Medical EvidencePLOS ONE, 2014
- Searching for research findings and KT literature2013
- Deploying an interactive machine learning system in an evidence-based practice center2012
- Machine learning in automated text categorizationACM Computing Surveys, 2002
- Support-vector networksMachine Learning, 1995