Large Language Models for the National Radiological Technologist Licensure Examination in Japan: Cross-Sectional Comparative Benchmarking and Evaluation of Model-Generated Items Study
Open Access
- 13 November 2025
- journal article
- research article
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 11, e81807
Abstract
Background: Mock examinations are widely used in health professional education to assess learning and prepare candidates for national licensure. However, instructor-written multiple-choice items can vary in difficulty, coverage, and clarity. Recently, large language models (LLMs) have achieved high accuracy in medical examinations, highlighting their potential for assisting item-bank development; however, their educational quality remains insufficiently characterized. Objective: This study aimed to (1) identify the most accurate LLM for the Japanese National Examination for Radiological Technologists and (2) use the top model to generate blueprint-aligned multiple-choice questions and evaluate their educational quality. Methods: Four LLMs—OpenAI o3, o4-mini, o4-mini-high (OpenAI), and Gemini 2.5 Flash (Google)—were evaluated on all 200 items of the 77th Japanese National Examination for Radiological Technologists in 2025. Accuracy was analyzed for overall items and for 173 nonimage items. The best-performing model (o3) then generated 192 original items across 14 subjects by matching the official blueprint (image-based items were excluded). Subject-matter experts (≥5 y as coordinators and routine mock examination authors) independently rated each generated item on five criteria using a 5-point scale (1=unacceptable, 5=adoptable): item difficulty, factual accuracy, accuracy of content coverage, appropriateness of wording, and instructional usefulness. Cochran Q with Bonferroni-adjusted McNemar tests compared model accuracies, and one-sided Wilcoxon signed-rank tests assessed whether the median ratings exceeded 4. Results: OpenAI o3 achieved the highest accuracy overall (90.0%; 95% CI 85.1%‐93.4%) and on nonimage items (92.5%; 95% CI 87.6%‐95.6%), significantly outperforming o4-mini on the full set ( P =.02). Across models, accuracy differences on the non-image subset were not significant (Cochran Q, P =.10). Using o3, the 192 generated items received high expert ratings for item difficulty (mean, 4.29; 95% CI 4.11‐4.46), factual accuracy (4.18; 95% CI 3.98‐4.38), and content coverage (4.73; 95% CI 4.60‐4.86). Ratings were comparatively lower for appropriateness of wording (3.92; 95% CI 3.73‐4.11) and instructional usefulness (3.60; 95% CI 3.41‐3.80). For these two criteria, the tests did not support a median rating >4 (one-sided Wilcoxon, P =.45 and P ≥.99, respectively). Representative low-rated examples (ratings 1‐2) and the rationale for those scores—such as ambiguous phrasing or generic explanations without linkage to stem cues—are provided in the supplementary materials. Conclusions: OpenAI o3 can generate radiological licensure items that align with national standards in terms of difficulty, factual correctness, and blueprint coverage. However, wording clarity and the pedagogical specificity of explanations were weaker and did not meet an adoptable threshold without further editorial refinement. These findings support a practical workflow in which LLMs draft syllabus-aligned items at scale, while faculty perform targeted edits to ensure clarity and formative feedback. Future studies should evaluate image-inclusive generation, use Application Programming Interface (API)-pinned model snapshots to increase reproducibility, and develop guidance to improve explanation quality for learner remediation.Keywords
This publication has 30 references indexed in Scilit:
- Differentiating between GPT-generated and human-written feedback for radiology residentsCurrent Problems in Diagnostic Radiology, 2025
- Pilot Study on Using Large Language Models for Educational Resource Development in Japanese Radiological Technologist ExamsMedical Science Educator, 2025
- MiniMedGPT: Efficient Large Vision–Language Model for medical Visual Question AnsweringPattern Recognition Letters, 2025
- Fusion Side Tuning: A Parameter and Memory Efficient Fine-tuning Method for High-resolution Medical Image Classification2024
- Potential of multimodal large language models for data mining of medical images and free-text reportsMeta-Radiology, 2024
- Leveraging large language models to construct feedback from medical multiple-choice QuestionsScientific Reports, 2024
- The performance of artificial intelligence-based large language models on ophthalmology-related questions in Swedish proficiency test for medicine: ChatGPT-4 omni vs Gemini 1.5 ProAJO International, 2024
- Large Language Models as Tools to Generate Radiology Board-Style Multiple-Choice QuestionsAcademic Radiology, 2024
- How Large Language Models Perform on the United States Medical Licensing Examination: A Systematic ReviewmedRxiv, 2023
- QUEST-AI: A System for Question Generation, Verification, and Refinement using AI for USMLE-Style ExamsmedRxiv, 2023