Xiao Wei

The future of artificial intelligence specialist doctors depends on foundational large language models (LLMs). However, there is currently a lack of evaluation studies on their Q&A capabilities in female malignancy. This cross-sectional study created a benchmark dataset consisting of 205 female malignancy Q&A to evaluate and compare the performance of 7 popular LLMs, with 3 human oncologists serving as a comparison. Only OpenAI o1, DeepSeek-R1, and Llama-3.1-405B achieved both accuracy and consistency rates exceeding 80%. Among these, Llama-3.1-405B demonstrated significantly higher accuracy than OpenAI o1 ( P = .002) and potentially higher than DeepSeek-R1 ( P = .059), with significantly lower latency ( P < .001). After removing token limits, both OpenAI o1 and DeepSeek-R1 exhibited strong reasoning abilities, achieving accuracy rates of 98.5% and 95.6%, respectively, with no significant difference from Llama-3.1-405B ( P > .05). The agreement between the models was fair, with an agreement rate of 0.318. Error analysis revealed that the highest error rates were associated with questions involving flexible medical scenarios and complex medical terminology. In female malignancy Q&A tasks, Llama-3.1-405B serves as a strong baseline model, while OpenAI o1 and DeepSeek-R1 show promise in inference, especially when token limitations are removed. However, their reasoning processes lack full consistency, likely due to differences in architectures and training methods. LLMs continue to face challenges in real-world medical scenarios and with complex medical terminology, highlighting the need for ongoing updates and the integration of clinical data to refine these models and improve their ability to address specialized medical needs.

Introduction: Malignant tumors represent a significant public health threat, and the integration of artificial intelligence in health care is increasingly becoming a priority. Many oncology institutions are already considering the use of DeepSeek-R1 to assist doctors in making complex medical decisions. However, there remains a lack of sufficient evidence regarding the accuracy, consistency, and cost-efficiency of DeepSeek-R1 and its distilled models in oncology decision-making. This study aims to fill this gap by evaluating the performance and cost-effectiveness of DeepSeek-R1 and its distilled models in oncology, providing critical insights into their potential for clinical integration. Objectives: This study aimed to systematically evaluate the performance, consistency, and cost-efficiency of the open-source large language model (LLM) DeepSeek-R1 and its distilled variants in the context of oncology decision-making, using a benchmark derived from the MedQA dataset. Methods: A custom oncology question set containing 1,206 multiple choice questions was curated from MedQA. Seven models, including DeepSeek-R1 and six distilled versions, were evaluated using an automated testing framework. Accuracy, consistency, latency, and token consumption were compared across models. Statistical tests, including McNemar and Wilcoxon signed-rank, were used to assess differences in performance. Questions were also categorized into clinical task types (diagnosis, treatment, triage, and follow-up) for subgroup analysis. Results: DeepSeek-R1 achieved the highest performance (accuracy: 91.38%; consistency: 90.47%), whereas DeepSeek-R1-Distill-Qwen-32B was the only distilled model to exceed both metrics at the 0.8 threshold (accuracy: 88.72%; consistency: 81.44%). DeepSeek-R1 demonstrated significantly higher accuracy than its distilled counterpart (p<0.05), particularly in diagnosis- and treatment-related tasks (p<0.05). However, it also exhibited significantly greater latency and token consumption. A Cohen’s kappa value of 0.575 indicated moderate agreement between the two models. Conclusion: DeepSeek-R1 is more suitable for high-stakes oncology tasks requiring high accuracy and consistency, whereas DeepSeek-R1-Distill-Qwen-32B offers a cost-effective alternative for use in outpatient or resource-limited settings. These findings support a task- and resource-adaptive deployment strategy for LLMs in clinical oncology.