Can ChatGPT evaluate research environments? Evidence from REF2021
Preprint
Abstract
UK academic departments are evaluated partly on the statements that they write about the value of their research environments for the Research Excellence Framework (REF) periodic assessments. These statements mix qualitative narratives and quantitative data, typically requiring time-consuming and difficult expert judgements to assess. This article investigates whether Large Language Models (LLMs) can support the process or validate the results, using the UK REF2021 unit-level environment statements as a test case. Based on prompts mimicking the REF guidelines, ChatGPT 4o-mini scores correlated positively with expert scores in almost all 34 (field-based) Units of Assessment (UoAs). ChatGPT's scores had moderate to strong positive Spearman correlations with REF expert scores in 32 out of 34 UoAs: 14 UoAs above 0.7 and a further 13 between 0.6 and 0.7. Only two UoAs had weak or no significant associations (Classics and Clinical Medicine). From further tests for UoA34, multiple LLMs had significant positive correlations with REF2021 environment scores (all p < .001), with ChatGPT 5 performing best (r=0.81; $ρ$=0.82), followed by ChatGPT-4o-mini (r=0.68; $ρ$=0.67) and Gemini Flash 2.5 (r=0.67; $ρ$=0.69). If LLM-generated scores for environment statements are used in future to help reduce workload, support more consistent interpretation, and complement human review then caution must be exercised because of the potential for biases, inaccuracy in some cases, and unwanted systemic effects. Even the strong correlations found here seem unlikely to be judged close enough to expert scores to fully delegate the assessment task to LLMs.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Research quality evaluation by AI in the era of large language models: advantages, disadvantages, and systemic effects – An opinion paperScientometrics, 2025
- Assessing the societal influence of academic research with ChatGPT: Impact case study evaluationsJournal of the Association for Information Science and Technology, 2025
- Evaluating the predictive capacity of ChatGPT for academic peer review outcomes across multiple platformsScientometrics, 2025
- Libraries as research enablers: A content analysis of research excellence framework environment statementsThe Journal of Academic Librarianship, 2025
- Can Large Language Models Provide Useful Feedback on Research Papers? A Large-Scale Empirical AnalysisNEJM AI, 2024
- What is a high-quality research environment? Evidence from the UK’s research excellence frameworkResearch Evaluation, 2024
- Artificial intelligence to support publishing and peer review: A summary and reviewLearned Publishing, 2023
- Investigating the relationship between research income and research excellence in education: Evidence from the REF2021‐UoA23 dataBritish Educational Research Journal, 2023
- Determinants of quality of research environment: An assessment of the environment submissions in the UK’s Research Excellence Framework in 2014Research Evaluation, 2020
- Semantic tone of research ‘environment’ submissions in the UK’s Research Evaluation Framework 2014Research Evaluation, 2017