Divergent LLM Adoption and Heterogeneous Convergence Paths in Research Writing
Preprint
Abstract
Large Language Models (LLMs), such as ChatGPT, are reshaping content creation and academic writing. This study investigates the impact of AI-assisted generative revisions on research manuscripts, focusing on heterogeneous adoption patterns and their influence on writing convergence. Leveraging a dataset of over 627,000 academic papers from arXiv, we develop a novel classification framework by fine-tuning prompt- and discipline-specific large language models to detect the style of ChatGPT-revised texts. Our findings reveal substantial disparities in LLM adoption across academic disciplines, gender, native language status, and career stage, alongside a rapid evolution in scholarly writing styles. Moreover, LLM usage enhances clarity, conciseness, and adherence to formal writing conventions, with improvements varying by revision type. Finally, a difference-in-differences analysis shows that while LLMs drive convergence in academic writing, early adopters, male researchers, non-native speakers, and junior scholars exhibit the most pronounced stylistic shifts, aligning their writing more closely with that of established researchers.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Promises and challenges of generative artificial intelligence for human learningNature Human Behaviour, 2024
- Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical AnalysisNEJM AI, 2024
- Generative AI and science communication in the physical sciencesNature Reviews Physics, 2024
- Factors associating with or predicting more cited or higher quality journal articles: An Annual Review of Information Science and Technology (ARIST) paperJournal of the Association for Information Science and Technology, 2023
- Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewersnpj Digital Medicine, 2023
- Abstracts written by ChatGPT fool scientistsNature, 2023
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity2023
- All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text2021
- A brief guide to the science and art of writing manuscripts in biomedicineJournal of Translational Medicine, 2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019