GenAI Content Detection Task 2: AI vs. Human -- Academic Essay Authenticity Challenge
Preprint
Abstract
This paper presents a comprehensive overview of the first edition of the Academic Essay Authenticity Challenge, organized as part of the GenAI Content Detection shared tasks collocated with COLING 2025. This challenge focuses on detecting machine-generated vs. human-authored essays for academic purposes. The task is defined as follows: "Given an essay, identify whether it is generated by a machine or authored by a human.'' The challenge involves two languages: English and Arabic. During the evaluation phase, 25 teams submitted systems for English and 21 teams for Arabic, reflecting substantial interest in the task. Finally, seven teams submitted system description papers. The majority of submissions utilized fine-tuned transformer-based models, with one team employing Large Language Models (LLMs) such as Llama 2 and Llama 3. This paper outlines the task formulation, details the dataset construction process, and explains the evaluation framework. Additionally, we present a summary of the approaches adopted by participating teams. Nearly all submitted systems outperformed the n-gram-based baseline, with the top-performing systems achieving F1 scores exceeding 0.98 for both languages, indicating significant progress in the detection of machine-generated text.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future DirectionsComputational Linguistics, 2025
- CHEAT: A Large-Scale Dataset for Detecting CHatGPT-writtEn AbsTractsIEEE Transactions on Big Data, 2025
- Simple techniques to bypass GenAI text detectors: implications for inclusive educationInternational Journal of Educational Technology in Higher Education, 2024
- Testing of detection tools for AI-generated textInternational Journal for Educational Integrity, 2023
- The ChatGPT conundrum: Human-generated scientific manuscripts misidentified as AI creations by AI text detection toolJournal of Pathology Informatics, 2023
- Distinguishing ChatGPT(-3.5, -4)-generated and human-written papers through Japanese stylometric analysisPLOS ONE, 2023
- GPT detectors are biased against non-native English writersPatterns, 2023
- Value attributed to text-based archives generated by artificial intelligenceRoyal Society Open Science, 2023
- Detecting and Unmasking AI-Generated Texts through Explainable Artificial Intelligence using Stylistic FeaturesInternational Journal of Advanced Computer Science and Applications, 2023
- DAWN2021