A crowdsourcing workflow for extracting chemical-induced disease relations from free text
Open Access
- 16 April 2016
- journal article
- research article
- Published by Oxford University Press (OUP) in Database: The Journal of Biological Databases and Curation
- Vol. 2016, 1-11
Abstract
Relations between chemicals and diseases are one of the most queried biomedical interactions. Although expert manual curation is the standard method for extracting these relations from the literature, it is expensive and impractical to apply to large numbers of documents, and therefore alternative methods are required. We describe here a crowdsourcing workflow for extracting chemical-induced disease relations from free text as part of the BioCreative V Chemical Disease Relation challenge. Five non-expert workers on the CrowdFlower platform were shown each potential chemical-induced disease relation highlighted in the original source text and asked to make binary judgments about whether the text supported the relation. Worker responses were aggregated through voting, and relations receiving four or more votes were predicted as true. On the official evaluation dataset of 500 PubMed abstracts, the crowd attained a 0.505 F-score (0.475 precision, 0.540 recall), with a maximum theoretical recall of 0.751 due to errors with named entity recognition. The total crowdsourcing cost was $1290.67 ($2.58 per abstract) and took a total of 7 h. A qualitative error analysis revealed that 46.66% of sampled errors were due to task limitations and gold standard errors, indicating that performance can still be improved. All code and results are publicly available at https://github.com/SuLab/crowd_cid_relex Database URL: https://github.com/SuLab/crowd_cid_relexKeywords
This publication has 31 references indexed in Scilit:
- DNorm: disease name normalization with pairwise learning to rankBioinformatics, 2013
- Web 2.0-Based Crowdsourcing for High-Quality Gold Standard Development in Clinical Natural Language ProcessingJournal of Medical Internet Research, 2013
- A CTD-Pfizer collaboration: manual curation of 88 000 scientific articles text mined for drug-disease and drug-phenotype interactionsDatabase: The Journal of Biological Databases and Curation, 2013
- Extraction of potential adverse drug events from medical case reportsJournal of Biomedical Semantics, 2012
- Promise of personalized omics to precision medicineWIREs Mechanisms of Disease, 2012
- Preparing for Precision MedicineNew England Journal of Medicine, 2012
- Using information mining of the medical literature to improve drug safetyJournal of the American Medical Informatics Association, 2011
- The curation paradigm and application tool used for manual curation of the scientific literature at the Comparative Toxicogenomics DatabaseDatabase: The Journal of Biological Databases and Curation, 2011
- Literature Mining for the Discovery of Hidden Connections between Drugs, Genes and DiseasesPLOS Computational Biology, 2010
- Unveiling the role of network and systems biology in drug discoveryTrends in Pharmacological Sciences, 2010