Mohamad Mukhtar Hamoud

Background: Large language models (LLMs) are seeing more use in clinical decision support and medical education, though how well they perform in arterial blood gas (ABG) interpretation is still not fully understood. Few studies have looked at whether model accuracy changes between English and non-English prompts. This study compared ABG interpretation performance of GPT-4o and Gemini 1.5 Pro in English and Turkish. Methods: We conducted a cross-sectional evaluation with 100 open ended questions covering ABG physiology, pathophysiology, and clinical application. Each question was presented to four configurations: GPT-4o and Gemini 1.5 Pro in English and Turkish. Two independent assessors scored responses using a 14-dimension rubric (0–2 points per dimension, maximum 28 points). We used Kruskal-Wallis and Friedman tests with Bonferroni correction for statistical comparisons, and Cohen’s kappa for inter-rater reliability. Hallucination rates and Winter formula accuracy served as secondary outcomes. Results: Mean total scores were: GPT-4o English 22.36 ± 2.11/28 (79.9%); GPT-4o Turkish 21.28 ± 2.54/28 (76.0%); Gemini 1.5 Pro English 19.81 ± 3.52/28 (70.7%); and Gemini 1.5 Pro Turkish 17.52 ± 3.26/28 (62.6%). Kruskal-Wallis H = 109.87, p < 0.001; all six pairwise comparisons remained significant after Bonferroni correction. All four models showed progressive score decline across parts (Friedman p < 0.001 each). English outperformed Turkish within both model families. Cohen’s kappa = 0.91. Hallucination rates were 4%, 7%, 18%, and 25%; Winter formula accuracy was 96%, 88%, 75%, and 63%, respectively. Conclusions: GPT-4o outperformed Gemini 1.5 Pro in both languages, and English prompts yielded higher scores than Turkish prompts. Performance deteriorated with increasing clinical complexity. These findings support continued human oversight for LLM use in ABG-related tasks, particularly in non-English settings and complex cases.