Coreference annotation and resolution in the Colorado Richly Annotated Full Text (CRAFT) corpus of biomedical journal articles
Authors:
- K. Bretonnel CohenK. Bretonnel Cohen
- Arrick LanfranchiALArrick Lanfranchi
- Miji Joo-Young ChoiMCMiji Joo-Young Choi
- Michael BadaMBMichael Bada
- William A. BaumgartnerWBWilliam A. Baumgartner
- Natalya PanteleyevaNPNatalya Panteleyeva
- Karin VerspoorKarin Verspoor
- Martha PalmerMPMartha Palmer
- Lawrence E. HunterLHLawrence E. Hunter
Open Access
- 17 August 2017
- journal article
- research article
- Published by Springer Nature in BMC Bioinformatics
- Vol. 18 (1) , 1-14
Abstract
Coreference resolution is the task of finding strings in text that have the same referent as other strings. Failures of coreference resolution are a common cause of false negatives in information extraction from the scientific literature. In order to better understand the nature of the phenomenon of coreference in biomedical publications and to increase performance on the task, we annotated the Colorado Richly Annotated Full Text (CRAFT) corpus with coreference relations. The corpus was manually annotated with coreference relations, including identity and appositives for all coreferring base noun phrases. The OntoNotes annotation guidelines, with minor adaptations, were used. Interannotator agreement ranges from 0.480 (entity-based CEAF) to 0.858 (Class-B3), depending on the metric that is used to assess it. The resulting corpus adds nearly 30,000 annotations to the previous release of the CRAFT corpus. Differences from related projects include a much broader definition of markables, connection to extensive annotation of several domain-relevant semantic classes, and connection to complete syntactic annotation. Tool performance was benchmarked on the data. A publicly available out-of-the-box, general-domain coreference resolution system achieved an F-measure of 0.14 (B3), while a simple domain-adapted rule-based system achieved an F-measure of 0.42. An ensemble of the two reached F of 0.46. Following the IDENTITY chains in the data would add 106,263 additional named entities in the full 97-paper corpus, for an increase of 76% percent in the semantic classes of the eight ontologies that have been annotated in earlier versions of the CRAFT corpus. The project produced a large data set for further investigation of coreference and coreference resolution in the scientific literature. The work raised issues in the phenomenon of reference in this domain and genre, and the paper proposes that many mentions that would be considered generic in the general domain are not generic in the biomedical domain due to their referents to specific classes in domain-specific ontologies. The comparison of the performance of a publicly available and well-understood coreference resolution system with a domain-adapted system produced results that are consistent with the notion that the requirements for successful coreference resolution in this genre are quite different from those of the general domain, and also suggest that the baseline performance difference is quite large.Keywords
Funding Information
- U.S. National Library of Medicine (G08 LM009639)
- U.S. National Library of Medicine (3G08 LM009639-02S1)
- U.S. National Library of Medicine (2R01LM009254)
- U.S. National Library of Medicine (R01LM008111)
- Australian Research Council (DP150101550)
- Defense Advanced Research Projects Agency (W911NF-14-C-0109)
This publication has 63 references indexed in Scilit:
- BioC: a minimalist approach to interoperability for biomedical text processingDatabase: The Journal of Biological Databases and Curation, 2013
- Concept annotation in the CRAFT corpusBMC Bioinformatics, 2012
- Boosting automatic event extraction from the literature using domain adaptation and coreference resolutionBioinformatics, 2012
- Anaphoric relations in the clinical narrative: corpus creationJournal of the American Medical Informatics Association, 2011
- Exploring subdomain variation in biomedical languageBMC Bioinformatics, 2011
- The structural and content aspects of abstracts versus bodies of full text journal articles are differentBMC Bioinformatics, 2010
- Extracting medical information from narrative patient records: the case of medication-related informationJournal of the American Medical Informatics Association, 2010
- Medication information extraction with linguistic pattern matching and semantic rulesJournal of the American Medical Informatics Association, 2010
- Evaluating contributions of natural language parsers to protein–protein interaction extractionBioinformatics, 2008
- Six senses in the literatureEMBO Reports, 2008