Creating and validating a scholarly knowledge graph using natural language processing and microtask crowdsourcing
Open Access
- 5 April 2023
- journal article
- research article
- Published by Springer Nature in International Journal on Digital Libraries
- Vol. 25 (2) , 273-285
Abstract
Due to the growing number of scholarly publications, finding relevant articles becomes increasingly difficult. Scholarly knowledge graphs can be used to organize the scholarly knowledge presented within those publications and represent them in machine-readable formats. Natural language processing (NLP) provides scalable methods to automatically extract knowledge from articles and populate scholarly knowledge graphs. However, NLP extraction is generally not sufficiently accurate and, thus, fails to generate high granularity quality data. In this work, we present TinyGenius, a methodology to validate NLP-extracted scholarly knowledge statements using microtasks performed with crowdsourcing. TinyGenius is employed to populate a paper-centric knowledge graph, using five distinct NLP methods. We extend our previous work of the TinyGenius methodology in various ways. Specifically, we discuss the NLP tasks in more detail and include an explanation of the data model. Moreover, we present a user evaluation where participants validate the generated NLP statements. The results indicate that employing microtasks for statement validation is a promising approach despite the varying participant agreement for different microtasks.Keywords
Funding Information
- European Research Council (819536)
This publication has 31 references indexed in Scilit:
- A Survey of Topic Modeling in Text MiningInternational Journal of Advanced Computer Science and Applications, 2015
- Studying Gamification: The Effect of Rewards and Incentives on Motivation2014
- WikidataCommunications of the ACM, 2014
- Crowdsourcing for bioinformaticsBioinformatics, 2013
- CrowdMap: Crowdsourcing Ontology Alignment with MicrotasksLecture Notes in Computer Science, 2012
- Article 50 million: an estimate of the number of scholarly articles in existenceLearned Publishing, 2010
- Answering the Call for a Standard Reliability Measure for Coding DataCommunication Methods and Measures, 2007
- Nasa-Task Load Index (NASA-TLX); 20 Years LaterProceedings of the Human Factors and Ergonomics Society Annual Meeting, 2006
- A study on tolerable waiting time: how long are Web users willing to wait?Behaviour & Information Technology, 2004
- Natural language processingAnnual Review of Information Science and Technology, 2003