Xinyue Zhang

Background: Unstructured electronic health records (EHRs) hinder the monitoring of intestinal infections. Large language models (LLMs) enable automated symptom extraction. However, their clinical validation is limited by a lack of systematic multimodel comparisons, unclear prompting strategies, and the privacy risks of cloud-based models (eg, data leakage and cross-border data transfer). Objective: This study aimed to systematically evaluate the performance of locally deployed open-source LLMs across 4 model families in extracting intestinal symptoms from unstructured EHR chief complaints under different prompting strategies. Methods: From a citywide health care information platform in Wuhan, China, we randomly selected 1000 chief complaints from outpatient records of intestinal clinics, infectious disease departments, pediatrics, and fever clinics. Six symptoms related to intestinal infectious diseases—diarrhea/bloody/mucoid stools, vomiting, abdominal pain, fever, nausea, and rash—were manually annotated as a gold-standard dataset. Twelve locally deployed open-source LLMs across 4 families, namely, Gemma3 (1b, 4b, 12b), Qwen3 (1.7b, 8b, 14b), DeepSeek-R1 (1.5b, 7b, 14b), and Llama (Llama2-Chinese 7b, 13b; Llama3.1 8b), were evaluated on the symptom extraction task using the gold-standard dataset. Three prompting strategies (no-role, zero-shot, and few-shot) were tested. Performance metrics included accuracy, precision, recall, F ₁ -score, specificity, balanced accuracy, and inference time. Statistical comparisons used Friedman tests for global differences, followed by Wilcoxon signed-rank and Mann-Whitney U tests with Bonferroni and false discovery rate corrections for pairwise comparisons. Results: Among the 4 families, Qwen3 models showed higher F ₁ -scores and balanced accuracy, with Qwen3-1.7b achieving a macroaveraged F ₁ -score of 0.85 under zero-shot prompting and Qwen3-8b reaching 0.89 under no-role prompting, while Gemma3 demonstrated robust performance at small to medium scales. Symptom-wise, models agreed more on frequent symptoms such as diarrhea and fever, whereas greater variability was observed for rarer symptoms like rash and nausea. The effect of prompting strategy varied across models, with no single strategy consistently outperforming the others. Although some pairwise differences reached statistical significance ( P <.05), the absolute gains in F ₁ -score were small. Conclusions: This study provides a systematic comparison of several open-source LLMs on a structured intestinal symptom extraction task. Among the LLM families, Qwen3 models offer a favorable balance between accuracy and efficiency, making them suitable for resource-constrained scenarios.

Background: Public health surveillance systems are critical for decision-making and have been advanced by monitoring infectious diseases. Objective: This study aims to assess the effectiveness and timeliness of multiple surveillance systems in tracking COVID-19 cases in the postpandemic era. Methods: Data of COVID-19–reported cases in a southern city of China were collected from the National Notifiable Disease Reporting Information System over a 1-year period, following the easing of the COVID-19 pandemic restrictions (from April 1, 2023, to June 30, 2024) as the operational benchmark. A total of 4 surveillance systems (hospital, wastewater, meteorological, and internet search engine) were integrated into a daily time series. Spearman correlation and 60-day moving window analyses with 7-day lags were used to assess associations. Distributed lag nonlinear models captured nonlinear meteorological effects. Time-series regression models assessed lead effects (0‐7 d) of each surveillance indicator, with and without meteorological adjustment. Results: Among 4 surveillance systems, 16 variables correlated significantly with reported cases. The nucleic acid amplification test (NAAT) positivity rate showed the strongest correlation, with a coefficient of 0.834 (95% CI 0.803‐0.860). Wastewater surveillance system demonstrated a moderate correlation, with the correlation coefficient of 0.776 (95% CI 0.737‐0.810) for the N gene positivity rate and 0.698 (95% CI 0.648‐0.743) for the N gene concentration. Moving-window analyses confirmed a stable correlation between NAAT positivity and reported cases (median 0.534, IQR 0.394‐0.724; 58% of windows ρ>0.5), while wastewater indicators exhibited greater temporal fluctuation, with the N gene concentration (median 0.585, IQR 0.214‐0.766; 60.8% of windows ρ>0.5) exceeding the N gene positivity rate (median 0.530, IQR 0.222‐0.742; 53.5% of windows ρ>0.5). Time-series analysis identified same-day associations (lag 0) for both NAAT positivity ( β =.819, 95% CI 0.768‐0.870) and wastewater signals (maximum effect: β =1.023, 95% CI 0.931‐1.115). Meteorological factors significantly modified the effect of internet surveillance indicators ( P <.05), particularly temperature and absolute humidity. Conclusions: An integrated, multichannel surveillance strategy of leveraging wastewater, clinical, and digital streams with meteorological contextualization can strengthen early warning and situational awareness for respiratory pathogen threats.