How Does a Generative Large Language Model Perform on Domain-Specific Information Extraction?─A Comparison between GPT-4 and a Rule-Based Method on Band Gap Extraction
- 8 October 2024
- journal article
- research article
- Published by American Chemical Society (ACS) in Journal of Chemical Information and Modeling
- Vol. 64 (20) , 7895-7904
Abstract
The advent of generative Large Language Models (LLMs) has greatly impacted the field of Natural Language Processing. However, it is inconclusive how generative LLMs perform on domain-specific information extraction tasks. This study compares the performance of GPT-4 and a rule-based information extraction method based on ChemDataExtractor on band gap information extraction, a task that has important implications for the materials science domain. No training data is required for either method, which is desirable because there is a lack of training data in the materials science domain compared with a variety of material information that is of interest. Manual evaluation on 415 randomly selected articles showed that the GPT-4 model achieved a higher level of accuracy in extracting materials' band gap information than the rule-based method (Correctness 87.95% vs 51.08%, Partial correctness 11.33% vs 36.87%, incorrectness 0.72% vs 12.05%). Further analysis of the errors reveals the strengths and weaknesses of the GPT-4 model compared to the rule-based method. The GPT-4 model shows stronger performance in interdependency resolution and complicated material name recognition, while it also has weaknesses in hallucination, identifying band gap values, and identifying band gap types. Revised prompt based on the error analysis leads to improved accuracy for GPT-4. To the best of our knowledge, this study is the first to compare the GPT-4 model and ChemDataExtractor for the band gap extraction task. This study provides evidence to support using generative LLMs for domain-specific information extraction tasks.Keywords
Funding Information
- Office of the Vice President for Research and Partnerships
- Research Council of the OU Norman Campus
- Data Institute for Societal Challenges
This publication has 36 references indexed in Scilit:
- Structured information extraction from scientific text with large language modelsNature Communications, 2024
- MechAgents: Large language model multi-agent collaborations can solve mechanics problems, generate new data, and integrate knowledgeExtreme Mechanics Letters, 2024
- Generative Retrieval-Augmented Ontologic Graph and Multiagent Strategies for Interpretive Large Language Model-Based Materials DesignACS Engineering Au, 2024
- BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio‐Inspired MaterialsAdvanced Science, 2023
- Zero-shot information extraction from radiological reports using ChatGPTInternational Journal of Medical Informatics, 2023
- Can ChatGPT be used to generate scientific hypotheses?Journal of Materiomics, 2023
- Deep neural network-based relation extraction: an overviewNeural Computing and Applications, 2022
- Opportunities and challenges of text mining in materials researchiScience, 2021
- Named Entity Recognition and Normalization Applied to Large-Scale Information Extraction from the Materials Science LiteratureJournal of Chemical Information and Modeling, 2019
- ChemDataExtractor: A Toolkit for Automated Extraction of Chemical Information from the Scientific LiteratureJournal of Chemical Information and Modeling, 2016