Validating LLM-Generated Relevance Labels for Educational Resource Search
Preprint
Abstract
Manual relevance judgements in Information Retrieval are costly and require expertise, driving interest in using Large Language Models (LLMs) for automatic assessment. While LLMs have shown promise in general web search scenarios, their effectiveness for evaluating domain-specific search results, such as educational resources, remains unexplored. To investigate different ways of including domain-specific criteria in LLM prompts for relevance judgement, we collected and released a dataset of 401 human relevance judgements from a user study involving teaching professionals performing search tasks related to lesson planning. We compared three approaches to structuring these prompts: a simple two-aspect evaluation baseline from prior work on using LLMs as relevance judges, a comprehensive 12-dimensional rubric derived from educational literature, and criteria directly informed by the study participants. Using domain-specific frameworks, LLMs achieved strong agreement with human judgements (Cohen's $κ$ up to 0.650), significantly outperforming the baseline approach. The participant-derived framework proved particularly robust, with GPT-3.5 achieving $κ$ scores of 0.639 and 0.613 for 10-dimension and 5-dimension versions respectively. System-level evaluation showed that LLM judgements reliably identified top-performing retrieval approaches (RBO scores 0.71-0.76) while maintaining reasonable discrimination between systems (RBO 0.52-0.56). These findings suggest that LLMs can effectively evaluate educational resources when prompted with domain-specific criteria, though performance varies with framework complexity and input structure.
Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Large Language Models can Accurately Predict Searcher Preferences2024
- Perspectives on Large Language Models for Relevance Judgment2023
- Retrieval of Educational Resources from the Web: A Comparison Between Google and Online Educational Repositories2019
- Supporting the discoverability of open educational resourcesEducation and Information Technologies, 2019
- Information retrieval in the workplace: A comparison of professional search practicesInformation Processing & Management, 2018
- Characterizing and Predicting the Multifaceted Nature of Quality in Educational Web ResourcesACM Transactions on Interactive Intelligent Systems, 2013
- The notion of relevance in teacher information behaviorProceedings of the American Society for Information Science and Technology, 2012
- Through efficient use of LORs: Prospective teachers' views on operational aspects of learning object repositoriesBritish Journal of Educational Technology, 2011
- Aspects of ‘relevance’ in the alignment of curriculum with educational standardsInformation Processing & Management, 2010
- Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevanceJournal of the American Society for Information Science and Technology, 2007