Assessing Generative Pre-trained Transformer 4 in Clinical Trial Inclusion Criteria Matching
- 9 June 2024
- report
- Published by Academy of Cognitive and Natural Sciences
Abstract
Clinical trials play a very important role in improving the quality and expectancy of life. At the same time, this is a complicated process which require a lot of time, effort, and costs. Effective interpretation and understanding of inclusion and exclusion criteria are key elements for streamlining trial documentation, enhancing patient engagement, and facilitating data analysis. The nuances in clinical criteria can significantly affect trial outcomes and participant selection across different linguistic and cultural backgrounds. The aim of this study is to assess the capability of the Generative Pre-trained Transformer 4 (GPT-4) model in comprehending and matching clinical trial inclusion criteria between English and Ukrainian, focusing on the model's potential to automate document processing and improve patient engagement in a multilingual context. The model was tested on 315 inclusion criteria from 15 clinical trials from ClinicalTrials.gov and their corresponding Ukrainian versions from ClinicalTrials.dec.gov.ua. The GPT-4 model accurately identified and matched 91% of the criteria pairs fully and demonstrated a detailed understanding of clinical language and nuances. An additional 4% of pairs were identified partially. All together it resulted in 95% accuracy for the tasks which accept partial matches. These findings indicate the model's substantial capacity for understanding complex medical and clinical terminologies and for facilitating the automation of clinical documents. Moreover, the experiment has already helped to identify several cases where the English and the Ukrainian versions had significant meaningful differences, most likely caused by human mistakes. The results of the study open a wide range of opportunities for automating clinical documentation processes and enhancing patient engagement through modern technologies. The scientific novelty is in applying Natural Language Processing (NLP) to understand and process clinical trial criteria in different languages, in this case Ukrainian, leading to more automated and patient-centered approaches in clinical research.Keywords
This publication has 20 references indexed in Scilit:
- The Future of Clinical TrialsJAMA, 2023
- Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final ExaminationScientific Reports, 2023
- GPT-4 Artificial Intelligence Model Outperforms ChatGPT, Medical Students, and Neurosurgery Residents on Neurosurgery Written Board-Like QuestionsWorld Neurosurgery, 2023
- Large language models in medicineNature Medicine, 2023
- GPT-4: a new era of artificial intelligence in medicineIrish Journal of Medical Science (1971 -), 2023
- Sentiment Analysis Technology of English Newspapers Quotes Based on Neural Network as Public Opinion Influences Identification Tool2022
- A scoping review of publicly available language tasks in clinical natural language processingJournal of the American Medical Informatics Association, 2022
- A systematic review of natural language processing for classification tasks in the field of incident reporting and adverse event analysisInternational Journal of Medical Informatics, 2019
- Natural Language Processing and Its Implications for the Future of Medication Safety: A Narrative Review of Recent Advances and ChallengesPharmacotherapy: The Journal of Human Pharmacology and Drug Therapy, 2018
- Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic reviewJournal of Biomedical Informatics, 2017