Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers
Preprint
Abstract
ChatGPT is a large language model developed by OpenAI. Despite its impressive performance across various tasks, no prior work has investigated its capability in the biomedical domain yet. To this end, this paper aims to evaluate the performance of ChatGPT on various benchmark biomedical tasks, such as relation extraction, document classification, question answering, and summarization. To the best of our knowledge, this is the first work that conducts an extensive evaluation of ChatGPT in the biomedical domain. Interestingly, we find based on our evaluation that in biomedical datasets that have smaller training sets, zero-shot ChatGPT even outperforms the state-of-the-art fine-tuned generative transformer models, such as BioGPT and BioBART. This suggests that ChatGPT's pre-training on large text corpora makes it quite specialized even in the biomedical domain. Our findings demonstrate that ChatGPT has the potential to be a valuable tool for various tasks in the biomedical domain that lack large annotated data.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Discovering drug–target interaction knowledge from biomedical literatureBioinformatics, 2022
- BioGPT: generative pre-trained transformer for biomedical text generation and miningBriefings in Bioinformatics, 2022
- Domain Adaptation with Pre-trained Transformers for Query Focused Abstractive Text SummarizationComputational Linguistics, 2022
- AMMU: A survey of transformer-based biomedical pretrained language modelsJournal of Biomedical Informatics, 2022
- A Primer in BERTology: What We Know About How BERT WorksTransactions of the Association for Computational Linguistics, 2020
- Question-driven summarization of answers to consumer health questionsScientific Data, 2020
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- BioCreative V CDR task corpus: a resource for chemical disease relation extractionDatabase: The Journal of Biological Databases and Curation, 2016
- Automatic semantic classification of scientific literature according to the hallmarks of cancerBioinformatics, 2015
- The DDI corpus: An annotated corpus with pharmacological substances and drug–drug interactionsJournal of Biomedical Informatics, 2013