Choosing an NLP Library for Analyzing Software Documentation: A Systematic Literature Review and a Series of Experiments
Authors:
- Fouad Nasser A Al OmranFOFouad Nasser A Al Omran
- Christoph TreudeCTChristoph Treude
- 1 May 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 187-197
Abstract
To uncover interesting and actionable information from natural language documents authored by software developers, many researchers rely on "out-of-the-box" NLP libraries. However, software artifacts written in natural language are different from other textual documents due to the technical language used. In this paper, we first analyze the state of the art through a systematic literature review in which we find that only a small minority of papers justify their choice of an NLP library. We then report on a series of experiments in which we applied four state-of-the-art NLP libraries to publicly available software artifacts from three different sources. Our results show low agreement between different libraries (only between 60% and 71% of tokens were assigned the same part-of-speech tag by all four libraries) as well as differences in accuracy depending on source: For example, spaCy achieved the best accuracy on Stack Overflow data with nearly 90% of tokens tagged correctly, while it was clearly outperformed by Google's SyntaxNet when parsing GitHub ReadMe files. Our work implies that researchers should make an informed decision about the particular NLP library they choose and that customizations to libraries might be necessary to achieve good results when analyzing software artifacts written in natural language.Keywords
This publication has 18 references indexed in Scilit:
- Automatic model generation from documentation for Java API functions2016
- Overcoming open source project entry barriers with a portal for newcomers2016
- Augmenting API documentation with insights from stack overflow2016
- Globally Normalized Transition-Based Neural Networks2016
- Synthesizing tests for detecting atomicity violations2015
- Cross-language program slicing for dynamic web applications2015
- TaskNav: Task-Based Navigation of Software Documentation2015
- Mining idioms from source code2014
- The Stanford CoreNLP Natural Language Processing Toolkit2014
- Facilitating Crowd Sourced Software Engineering via Stack Overflow2013