Assessing the Reproducibility of the Structured Abstracts Generated by ChatGPT and Bard Compared to Human-Written Abstracts in the Field of Spine Surgery: Comparative Analysis
Authors:
- Hong Jin KimHKHong Jin Kim
- Jae Hyuk YangJYJae Hyuk Yang
- Dong-Gune ChangDong-Gune Chang
- Lawrence G LenkeLLLawrence G Lenke
- Javier PizonesJPJavier Pizones
- René CasteleinRené Castelein
- Kota WatanabeKWKota Watanabe
- Per D TrobischPTPer D Trobisch
- Gregory M Mundis JrGJGregory M Mundis Jr
- Seung Woo SuhSSSeung Woo Suh
- Se-Il SukSSSe-Il Suk
Open Access
- 26 June 2024
- journal article
- research article
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26 (1) , e52001
Abstract
Background: Due to recent advances in artificial intelligence (AI), language model applications can generate logical text output that is difficult to distinguish from human writing. ChatGPT (OpenAI) and Bard (subsequently rebranded as “Gemini”; Google AI) were developed using distinct approaches, but little has been studied about the difference in their capability to generate the The use of AI to write scientific abstracts in the field of spine surgery is the center of much debate and controversy. Objective: The objective of this study is to assess the reproducibility of the structured abstracts generated by ChatGPT and Bard compared to human-written abstracts in the field of spine surgery. Methods: In total, 60 abstracts dealing with spine sections were randomly selected from 7 reputable journals and used as ChatGPT and Bard input statements to generate abstracts based on supplied paper titles. A total of 174 abstracts, divided into human-written abstracts, ChatGPT-generated abstracts, and Bard-generated abstracts, were evaluated for compliance with the structured format of journal guidelines and consistency of content. The likelihood of plagiarism and AI output was assessed using the iThenticate and ZeroGPT programs, respectively. A total of 8 reviewers in the spinal field evaluated 30 randomly extracted abstracts to determine whether they were produced by AI or human authors. Results: The proportion of abstracts that met journal formatting guidelines was greater among ChatGPT abstracts (34/60, 56.6%) compared with those generated by Bard (6/54, 11.1%; P<.001). However, a higher proportion of Bard abstracts (49/54, 90.7%) had word counts that met journal guidelines compared with ChatGPT abstracts (30/60, 50%; P<.001). The similarity index was significantly lower among ChatGPT-generated abstracts (20.7%) compared with Bard-generated abstracts (32.1%; P<.001). The AI-detection program predicted that 21.7% (13/60) of the human group, 63.3% (38/60) of the ChatGPT group, and 87% (47/54) of the Bard group were possibly generated by AI, with an area under the curve value of 0.863 (P<.001). The mean detection rate by human reviewers was 53.8% (SD 11.2%), achieving a sensitivity of 56.3% and a specificity of 48.4%. A total of 56.3% (63/112) of the actual human-written abstracts and 55.9% (62/128) of AI-generated abstracts were recognized as human-written and AI-generated by human reviewers, respectively. Conclusions: Both ChatGPT and Bard can be used to help write abstracts, but most AI-generated abstracts are currently considered unethical due to high plagiarism and AI-detection rates. ChatGPT-generated abstracts appear to be superior to Bard-generated abstracts in meeting journal formatting guidelines. Because humans are unable to accurately distinguish abstracts written by humans from those produced by AI programs, it is crucial to exercise special caution and examine the ethical boundaries of using AI programs, including ChatGPT and Bard.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Quantitative evaluation of ChatGPT versus Bard responses to anaesthesia-related queriesBritish Journal of Anaesthesia, 2023
- ChatGPT, Bard, and Large Language Models for Biomedical Research: Opportunities and PitfallsAnnals of Biomedical Engineering, 2023
- Chat Generative Pre-trained Transformer–written obstetrics and gynecology abstracts fool practitionersAmerican Journal of Obstetrics & Gynecology MFM, 2023
- Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewersnpj Digital Medicine, 2023
- Will ChatGPT transform healthcare?Nature Medicine, 2023
- Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research ScenariosJournal of Medical Systems, 2023
- ChatGPT: five priorities for researchNature, 2023
- Large language models generate functional protein sequences across diverse familiesNature Biotechnology, 2023
- Abstracts written by ChatGPT fool scientistsNature, 2023
- A computational approach to measure the linguistic characteristics of psychotherapy timing, responsiveness, and consistencynpj Mental Health Research, 2022