Text mining methods for automated data extraction from health technology assessment reports of medicines using classical natural language processing and generative artificial intelligence
Open Access
- 6 March 2026
- journal article
- Published by Oxford University Press (OUP) in JAMIA Open
- Vol. 9 (2)
Abstract
Objective: This proof of concept for utilizing automatic data extraction methods to extract health technology assessment (HTA) attributes from HTA reports of medicines aimed to explore which attributes could be extracted and how accurately, using different data extraction methods. This enables easy access to insights into HTA recommendations for policymaking and policy-related research. Materials and Methods: In total, 14 relevant attributes (eg, assessment outcome or date) were identified for extraction using two classical natural language processing (NLP) methods (rule-based and classification models) and a generative AI method (large language model (LLM)-based, i.e., Claude 3 Opus). The performance of these techniques was compared using 50 HTA reports published by the National Institute for Health and Care Excellence (NICE, United Kingdom). Results: All three methods were able to extract certain attributes with high accuracy, with differences between the extraction methods and the type of attribute. The LLM-based extraction was the only method able to extract attributes on a medicine-indication combination level. The LLM-based extraction performed best (88–98% semantical accuracy for 12/14 attributes). Extraction of Outcome relative effectiveness analyses (REA) and Comparator was the most challenging and had the lowest accuracy (∼70% for the LLM-based extraction). Discussion & Conclusion: Automatic data extraction for relevant attributes from HTA reports is possible, but there is still room for improvement. LLM-based extraction outperformed the two NLP methods, but challenges regarding the use of commercial software and reproducibility remain. Future research should focus on expanding the system to other HTA organizations and further refining the LLM-based extraction.Keywords
This publication has 22 references indexed in Scilit:
- Added benefit and revenues of oncology drugs approved by the European Medicines Agency between 1995 and 2020: retrospective cohort studyBMJ, 2024
- Barriers to Use Artificial Intelligence Methodologies in Health Technology Assessment in Central and East European CountriesFrontiers in Public Health, 2022
- A Review on Human–AI Interaction in Machine Learning and Insights for Medical ApplicationsInternational Journal of Environmental Research and Public Health, 2021
- The new definition of health technology assessment: A milestone in international collaborationInternational Journal of Technology Assessment in Health Care, 2020
- Error rates of human reviewers during abstract screening in systematic reviewsPLOS ONE, 2020
- Differences in Health Technology Assessment Recommendations Among European Jurisdictions: The Role of Practice VariationsValue in Health, 2020
- Weighing of Evidence by Health Technology Assessment Bodies: Retrospective Study of Reimbursement Recommendations for Conditionally Approved DrugsClinical Pharmacology & Therapeutics, 2018
- Policies for Use of Real-World Data in Health Technology Assessment (HTA): A Comparative Study of Six HTA AgenciesValue in Health, 2017
- Health Technology Assessment (HTA) Case Studies: Factors Influencing Divergent HTA Reimbursement Recommendations in Australia, Canada, England, and ScotlandValue in Health, 2017
- Commonalities and differences in HTA outcomes: A comparative analysis of five countries and implications for coverage decisionsHealth Policy, 2012