Validating AI-assisted evaluation of open science practices in brain sciences: ChatGPT, Claude and human expert comparisons
Open Access
- 18 February 2026
- journal article
- Published by The Royal Society in Royal Society Open Science
- Vol. 13 (2)
Abstract
This study investigates the efficacy of AI-assisted evaluation of open science practices in brain sciences, comparing ChatGPT 4 and Claude 3.5 Sonnet against human expert assessment. We analysed 100 randomly selected journal articles across various brain science disciplines using a six-item transparency checklist. Three human experts and two AI chatbots independently evaluated the articles. Intraclass correlation analyses revealed moderate-to-good agreement between human consensus scores and AI evaluations, with pairwise correlations providing convergent evidence of substantial correspondence. Both chatbots demonstrated high concordance with humans in assessing code sharing, materials availability, preregistration and sample size rationales. However, they struggled with accurately identifying the presence of data availability statements and assessing public accessibility of shared data. Importantly, AI assessments were restricted to information stated within the article; external links or repository contents were not cross-checked owing to limited web search capability of AI chatbots. These findings suggest that AI chatbots can effectively support the evaluation of some open science practices and potentially expedite the assessment process in academic research. However, their limitations in certain areas highlight the continued importance of human oversight in ensuring comprehensive and accurate evaluations of scientific transparency.Keywords
Funding Information
- UCL Research Culture
- United Kingdom Economic and Social Research Council
This publication has 40 references indexed in Scilit:
- Beyond principlism: practical strategies for ethical AI use in research practicesAI and Ethics, 2024
- Human bias in AI models? Anchoring effects and mitigation strategies in large language modelsJournal of Behavioral and Experimental Finance, 2024
- Sharing practices of software artefacts and source code for reproducible researchInternational Journal of Data Science and Analytics, 2024
- Performance of a Large Language Model in Screening CitationsJAMA Network Open, 2024
- Raw data were not disclosed in 95% of PubMed-indexed heart failure meta-analyses in 2021: A systematic analysis of transparencyInternational Journal of Cardiology, 2024
- Open scienceCurrent Biology, 2023
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- The Rigor and Transparency Index Quality Metric for Assessing Biological and Medical Science MethodsiScience, 2020
- A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability ResearchJournal of Chiropractic Medicine, 2016
- Open Science: One Term, Five Schools of Thought2013