Cognitively Diverse Multiple-Choice Question Generation: A Hybrid Multi-Agent Framework with Large Language Models
Preprint
- 3 February 2026
- preprint
- Published by MDPI AG in Preprints.org
Abstract
Recent advances in large language models (LLMs) have made automated multiple-choice question (MCQ) generation increasingly feasible; however, reliably producing items that satisfy controlled cognitive demands remains a challenge. To address this gap, we introduce ReQUESTA, a hybrid, multi-agent framework for generating cognitively diverse MCQs that systematically target text-based, inferential, and main idea comprehension. ReQUESTA decomposes MCQ authoring into specialized subtasks and coordinates LLM-powered agents with rule-based components to support planning, controlled generation, iterative evaluation, and post-processing. We evaluated the framework in a large-scale reading comprehension study using academic expository texts, comparing ReQUESTA-generated MCQs with those produced by a single-pass GPT-5 zero-shot baseline. Psychometric analyses of learner responses assessed item difficulty and discrimination, while expert raters evaluated question quality across multiple dimensions, including topic relevance and distractor quality. Results showed that ReQUESTA-generated items were consistently more challenging, more discriminative, and more strongly aligned with overall reading comprehension performance. Expert evaluations further indicated stronger alignment with central concepts and superior distractor linguistic consistency and semantic plausibility, particularly for inferential questions. These findings demonstrate that hybrid, agentic orchestration can systematically improve the reliability and controllability of LLM-based generation, highlighting workflow design as a key lever for structured artifact generation beyond single-pass prompting.Keywords
All Related Versions
- Version 1, 2026-02-03, Preprints
- Published version: Electronics, 15 (6), 1209.
This publication has 22 references indexed in Scilit:
- ChatGPT as an automated essay scoring tool in the writing classrooms: how it compares with human scoringEducation and Information Technologies, 2024
- Automatic question-answer pairs generation using pre-trained large language models in higher educationComputers and Education: Artificial Intelligence, 2024
- Exploring quality criteria and evaluation methods in automated question generation: A comprehensive surveyEducation and Information Technologies, 2024
- Applying large language models and chain-of-thought for automatic scoringComputers and Education: Artificial Intelligence, 2024
- Automated Question and Answer Generation from Texts using Text-to-Text TransformersArabian Journal for Science and Engineering, 2023
- A comprehensive review on MCQ generation from textMultimedia Tools and Applications, 2023
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- A Systematic Review of Automatic Question Generation for Educational PurposesInternational Journal of Artificial Intelligence in Education, 2019
- Multiple-choice testing in education: Are the best practices for assessment also good for learning?Journal of Applied Research in Memory and Cognition, 2018
- A computer-aided environment for generating multiple-choice test itemsNatural Language Engineering, 2006