SPShaowen Peng

Shaowen Peng

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Shaowen Peng has a total of 14 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Concept-Aware Learning Framework for Detecting Common Symptoms in Social Media Text: Model Development and Evaluation Study

    by Michael Van Supranes, Shaowen Peng, Tomohiro Nishiyama, Shoko Wakamiya, Eiji Aramaki

    Background: Social media offers valuable information for public health surveillance, but detecting valid symptom expressions in noisy and informal text remains challenging. Fine-tuned language models can achieve strong test accuracy while relying on spurious patterns, limiting their reliability in real-world applications. Objective: This study aimed to develop a more robust symptom detection model by incorporating surveillance-relevant attributes into the training objective and introducing a counterfactual-based evaluation framework to assess attribute sensitivity. Methods: We proposed a concept-aware learning framework (CALF), which combines multiattribute counterfactual data augmentation with multitask learning. We compared standard fine-tuning (baseline) with data-level interventions, a model-level intervention, hybrid strategies, and prompt-based methods across multiple encoders and LLMs for data augmentation. Performance was evaluated on the following 3 datasets: the MedWeb test using macro F1-score (m-F1), the MedWeb checklist using flip accuracy (FA), and a supplementary test set using F1-score for flu-or-fever detection (f-F1). FA measures sensitivity to surveillance-relevant attributes, and the supplementary test score serves as an indicator for out-of-distribution performance. Results: The baseline achieved an m-F1-score of 0.871 on the test set but a low overall FA on the checklist (FA=0.342). It also has limited sensitivity to recovery status, timing of occurrence, and uncertainty of expression. Compared with the baseline, CALF has a comparable MedWeb test score (m-F1=0.880 vs 0.871; P=.18), but a substantially higher FA (FA=0.830 vs 0.342; P=.001) and supplementary test score (f-F1=0.856 vs 0.797; P=.007). CALF also improved sensitivity to recovery, timing, and uncertainty across pretrained encoders. The gains were generally larger when GPT-4.1 (OpenAI) was used for data augmentation. Under GPT-4.1, CALF often achieved the highest overall FA among encoder-based methods. When Qwen2.5-7B-Instruct (Alibaba Cloud) was used, an alternative hybrid strategy may achieve comparable or slightly better results. Checklist performance was strongly associated with supplementary test performance (ρ=0.811), whereas MedWeb test performance showed weak association with supplementary test performance (ρ=–0.042). Prompting GPT-4.1 achieved the highest FA overall (FA=0.920) and the highest supplementary test score (f-F1=0.910), but a lower test set score than the baseline (m-F1=0.776 vs 0.871). Conclusions: Standard held-out metrics alone do not adequately characterize robustness to dataset-specific or spuriously correlated features. Models may perform well on the test set while remaining insensitive to attributes that influence whether a symptom mention is relevant to surveillance. The MedWeb checklist complements conventional evaluation by measuring sensitivity to controlled attribute changes and may indicate out-of-distribution performance. CALF improved sensitivity to surveillance-relevant attributes while maintaining strong classification performance, supporting concept-aware learning as a practical strategy for developing more robust encoder-based models.

  • Open Access

    Multilingual Disparities in Large Language Model–Based Symptom Detection for Global Disease Surveillance: Evaluation Study

    by Sa'IDah Zahrotul Jannah, Tomohiro Nishiyama, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

    Background: Symptom detection is essential in global disease surveillance to detect potential outbreaks, as symptoms are the first observable signs of infection. To reflect real-time ground truth conditions during pandemics, social media has emerged as a valuable data source. Moreover, effective digital disease surveillance systems must operate across diverse linguistic settings, and large language models (LLMs) have been shown to perform inconsistently across languages, tending to have lower performance in low-resource languages. While multilingual approaches have been explored in various health-related natural language processing tasks, a critical gap remains in understanding whether LLM-based symptom detection can perform consistently across languages for global disease surveillance. Southeast Asia demonstrates this challenge, combining diverse languages and the potential for emerging infectious disease outbreaks, making it a case for evaluating how multilingual performance disparities manifest in symptom detection. Objective: This study aims to evaluate multilingual disparities in symptom detection using a LLM, as well as the associated error mechanisms across languages and symptom types, to better understand their implications for global disease surveillance. Methods: This study uses the MedWeb dataset, a multilingual pseudo–social media text dataset with multiple symptom labels. The data consist of 12 languages, covering diverse regions and language resource classifications. The symptoms included in this study are fever, headache, runny nose, cough, diarrhea, hay fever, influenza, and cold. We used GPT-5 as the symptom detection system, representing a strong model for health-related tasks. The results are evaluated using the macro precision, recall, and F 1 -score. An error analysis was conducted to identify the error mechanisms underlying incorrect symptom predictions across languages and to calculate the impact of each error category. Results: Our findings show that performance varied across languages, language resource groups, and symptom types. Southeast Asian (SEA) languages generally achieved lower scores than non-SEA languages, with Japanese obtaining the highest F 1 -score (0.812) and Lao the lowest (0.716). High-resource languages achieved the most consistent performance, while low-resource languages obtained the lowest overall scores. At the symptom level, diarrhea, headache, cough, and fever showed stable detection across languages, while hay fever, runny nose, influenza, and cold exhibited greater variability. Hay fever showed the widest variability, forming 2 distinct performance clusters aligned with language resource and region classifications. The error analysis revealed 4 misclassification patterns: explicitly mentioned symptoms, cross-lingual variation, symptom overgeneralization, and context misinterpretation. Cross-lingual variation was the most frequent error category, while errors related to explicitly mentioned symptoms showed the largest potential impact on model performance. Conclusions: Due to performance disparities, achieving more reliable and equitable LLM-based symptom detection from social media text for global disease surveillance would benefit from broader representation in training data for low-resource languages, improved cultural and linguistic sensitivity, and stronger contextual understanding of symptom-related expressions.

  • Open Access

    RED: Retrieval-enhanced knowledge distillation for smaller language models

    by Xinbai Li, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

  • Open Access

    Multilingual Disparities in LLM-Based Symptom Detection for Global Disease Surveillance: Findings from High- and Low-Resource Languages (Preprint)

    by Sa'IDah Zahrotul Jannah, Tomohiro Nishiyama, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

    BACKGROUND Symptom detection is essential in global disease surveillance to detect potential outbreaks, as symptoms are the first observable signs of infection. To reflect real-time ground truth conditions during pandemics, social media has emerged as a valuable data source. Moreover, effective digital disease surveillance systems must operate across diverse linguistic settings, but large language models (LLMs) have been shown to perform inconsistently across languages, with a tendency to have lower performance on low-resource languages. While multilingual approaches have been explored in various health-related NLP tasks, a critical gap remains in understanding whether LLM-based symptom detection can perform consistently across languages for global disease surveillance. Southeast Asia demonstrates this challenge, combining diverse languages and the potential for emerging infectious disease outbreaks, making it a case for evaluating how multilingual performance disparities manifest in symptom detection. OBJECTIVE This study aims to evaluate multilingual disparities in symptom detection using a large language model across languages and symptom types to better understand their implications for global disease surveillance. METHODS This study uses the MedWeb dataset, a multilingual pseudo–social media text dataset with multiple symptom labels. The data consist of 12 languages, covering diverse regions and language resource classifications. The symptoms included in this study are fever, headache, runny nose, cough, diarrhea, hay fever, influenza, and cold. We employed GPT-5 as the symptom detection system, representing a strong model for health-related tasks. The results are evaluated using the macro precision, recall, and F1-score. RESULTS Our findings show that performance varied across languages, language resource groups, and symptom types. Southeast Asian (SEA) languages generally achieved lower scores than non-SEA, with Japanese obtaining the highest F1-score (0.812) and Lao the lowest (0.716). High-resource languages achieved the most consistent performance, while low-resource languages obtained the lowest overall scores. At the symptom level, Diarrhea, Headache, Cough, and Fever showed stable detection across languages, while Hay fever, Runny nose, Influenza, and Cold exhibited greater variability. Hay fever showed the widest variability, forming two distinct performance clusters, aligned with language resource and region classification. The error analysis revealed four misclassification patterns: explicitly mentioned symptoms, cross-lingual variation, symptom overgeneralization, and context misinterpretation. CONCLUSIONS Due to the performance disparities, to achieve more reliable and equitable LLM-based symptom detection from social media text for global disease surveillance, it would benefit from broader representation of training data for low-resource languages, improved cultural-linguistic sensitivity, and stronger contextual understanding of symptom-related expressions.

  • Open Access

    A Multiassessment and Multiprofessional Agents Approach for Medical Chatbot Risk Estimation: Development and Evaluation Study

    by Lenard Paulo Velasco Tamayo, Tomohiro Nishiyama, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

    Background: Assessing chatbot responses across 3 domains—medical, ethical, and legal—is essential to ensuring the safe use of artificial intelligence in health care. Although advancements in the use of large language models (LLMs) show significant improvements in evaluating question-answer datasets, such as multiple-choice medical exams, existing systems use general LLMs without incorporating specialized domain knowledge. They rely on standardized instructions without integrating real-world information, and ensemble methods such as majority voting fail to resolve disagreements among agents, resulting in misclassification and challenges in risk assessment. Objective: This study aims to design, develop, and evaluate a synergistic approach for assessing risks associated with chatbot responses using multiassessment (MA) and multiprofessional agents (MPAs). Methods: We designed and developed an approach consisting of MA and MPA, specifically initial assessment (MA1), which internalizes 3 roles and provides an initial risk estimation, and final assessment (MA3), which aims to reach a final consensus based on the previous assessments (MA1 and MA2), with each using 1 LLM. The verification assessment (MA2) incorporates an MPA or role-based LLM specialized agents for each risk domain (medical, ethical, and legal). We evaluated the proposed approach using the MedNLP-CHAT (Medical Natural Language Processing for AI Chat) corpus (N=226; 100 train, 126 test), covering baseline, enhanced prompt, embedding-based search, and retrieval-augmented generation (RAG). Primary metrics included macro F1-score and joint accuracy to evaluate system performance, along with CI and paired macro F1-score difference (Δ) as supporting metrics to assess the approach’s effectiveness. Results: The MA-MPA framework integrated with RAG achieved the highest average macro F1-score of 0.800 across risk domains and a joint accuracy of 76 (60.3%) correct predictions across all risk domains out of 126 question-answer pairs, with notable improvements over the best reported Eighteenth NII Testbeds and Community for Information Access Research Project (NTCIR-18) MedNLP-CHAT systems in the ethical (+0.252) and legal (+0.096) risk domains, while the medical domain showed a modest increase of +0.070. The MA approach contributed the largest gains, particularly from MA1 to MA2, with paired macro F1-score gains ranging from +0.176 to +0.214 across systems. The MPA approach performed better when integrated with MA and external knowledge, with paired bootstrap estimates showing a gain of +0.037 (95% CI 0.003-0.074) over baseline; however, joint accuracy gains were not evident (95% CI –2.9% to 7.7%), and gains relative to the enhanced prompt were small. Notably, MA alone achieved higher joint accuracy than RAG (62.7% vs 60.3%), indicating a metric-specific trade-off rather than consistent superiority across all metrics. Conclusions: The MA-MPA approach shows potential for improving risk estimation in chatbot responses. The results suggest that the framework is particularly useful for enhancing balanced overall performance, especially when combined with external knowledge, although the medical risk domain remains challenging. Furthermore, more specialized LLMs may further improve contextually grounded risk estimation.


  • Open Access

    Estimating Shared Mental Models via Communication-Categorized Directed Graphs

    by Hiroaki Tanaka, Wataru Yamada, Keiichi Ochiai, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

  • Open Access

    Robust Detection of Common Symptoms in Social Media Text using Concept-Aware Learning Framework (Preprint)

    by Michael Van Supranes, Shaowen Peng, Tomohiro Nishiyama, Shoko Wakamiya, Eiji Aramaki

    BACKGROUND Social media can provide valuable information for public health surveillance, but detecting valid symptom expressions in noisy and informal text remains challenging. While fine-tuned language models can achieve strong test-set accuracy, they are prone to relying on spurious patterns rather than surveillance-relevant features, which limits their reliability in real-world applications. OBJECTIVE This study aims to develop a more robust symptom detection model by explicitly incorporating surveillance-relevant attributes into the training objective and by introducing a counterfactual-based evaluation framework to better assess robustness. METHODS We propose the Concept-Aware Learning Framework (CALF). This strategy allows one to identify a set of surveillance-relevant concepts, and incorporate it in the fine-tuning process. CALF does this by integrating multi-attribute counterfactual data augmentation with a multi-task learning objective. Different model fine-tuning strategies were compared on a multi-label symptom classification dataset, and model robustness was further probed using a newly designed counterfactual-based test. RESULTS All training strategies achieved comparable macro F1 scores on the standard test set (range: 0.84-0.89). However, substantial differences emerged under the counterfactual-based test. Based on the counterfactual-based test, CALF achieved the highest overall flip accuracy (0.82; 95% CI: 0.80–0.84), significantly outperforming standard supervised fine-tuning (0.45; 95% CI: 0.43–0.47), data-augmentation-only approaches, and alternative hybrid methods. While the baseline model was primarily sensitive to direct symptom mentions, CALF substantially improved sensitivity to other surveillance-relevant concepts, including recovery status, timing of occurrence, and uncertainty. Robustness gains were also consistent across most symptom categories and pretrained models, and qualitative error analysis showed that CALF was less likely to misclassify historical, speculative, or informational mentions as positive cases. CONCLUSIONS High test-set accuracy is not a good indicator of model robustness. Standard supervised fine-tuning performed well on the test set, but the counterfactual-based test implies that it is not robust to data-specific biases. In contrast, CALF maintained good performances in both the test set and the counterfactual-based test. This indicates CALF allows one to train a model that is both accurate and sensitive to surveillance-relevant concepts.

  • Open Access

    Persona-Driven Data Augmentation for Disease Name Recognition Across Rare and General Disease Corpora: Comparative Evaluation Study (Preprint)

    by Jude Crener Junior Pierre, Tomohiro Nishiyama, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

    BACKGROUND Medical information extraction requires automatically identifying disease names and related terms in text. This task, known as named entity recognition (NER), relies on expert-annotated data that are costly to produce and often available only in limited quantities. Data augmentation (DA) aims to expand available training data; however, standard techniques such as synonym replacement and back-translation may introduce inappropriate substitutions or fail to preserve entity-label alignment, which is critical for sequence-labeling tasks. Although large language models can generate fluent text, their outputs may also contain factual inconsistencies or unintended changes if not carefully controlled. OBJECTIVE This study investigated whether persona-driven, document-level DA using a large language model could improve biomedical disease NER performance by generating diverse rephrasings of medical documents while preserving annotated entities. METHODS We designed a DA framework using multiple personas that varied in medical expertise, personality, tone, and narrative style. Using prompting constrained by XML tags, each persona rephrased training documents while aiming to preserve annotated entity spans. We evaluated the framework on 2 biomedical disease NER datasets with complementary roles: RareDis, a low-resource rare disease corpus, and National Center for Biotechnology Information (NCBI) disease, a more general disease benchmark. Semantic fidelity and lexical diversity were measured using BERTScore and Bilingual Evaluation Understudy (BLEU-4), respectively, and personas were grouped into high-, balanced-, and low-fidelity subsets. Biomedical pretrained BioBERT models were fine-tuned and evaluated under multiple settings, including gold-standard (GS) data only, synonym replacement, single-persona augmentation, curated persona subsets, and all-persona augmentation. Performance was assessed using microaveraged entity-level precision, recall, and F1-score, and results were examined at both the overall and individual entity-type levels. Performance values are reported as mean (SD). RESULTS Persona-driven DA improved NER performance over GS-only training in both datasets, with the strongest gains obtained by combining multiple persona-generated variants with GS data. In RareDis, the best result was achieved by the low-fidelity subset (mean F1-score 73.35, SD 0.19 vs baseline 71.22, SD 0.45), while in NCBI disease, the all-personas setting performed best (mean F1-score 89.32, SD 0.26 vs baseline 87.82, SD 0.18). In low-resource experiments, the all-personas and high-fidelity persona settings in NCBI disease exceeded the performance of the model trained on 100% GS data using only 60% of the training data, whereas gains in RareDis were more modest. Entity-level analysis showed improvements across RareDis categories, particularly for symptom, and confusion analysis indicated reduced symptom-sign confusion under augmentation. CONCLUSIONS Persona-driven DA improved biomedical disease NER by introducing controlled linguistic variation while largely preserving annotated entities. The strongest gains were obtained when multiple persona-generated variants were combined with GS data, although the benefit varied across datasets. These findings suggest that this approach is a promising strategy for low-resource biomedical NER.

  • Open Access

    Persona-Driven Data Augmentation for Disease Name Recognition Across Rare and General Disease Corpora: A Comparative Evaluation (Preprint)

    by Jude Crener Junior Pierre, Tomohiro Nishiyama, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

    Background: Medical information extraction requires automatically identifying disease names and related terms in text. This task, known as named entity recognition (NER), relies on expert-annotated data that are costly to produce and often available only in limited quantities. Data augmentation (DA) aims to expand available training data; however, standard techniques such as synonym replacement and back-translation may introduce inappropriate substitutions or fail to preserve entity-label alignment, which is critical for sequence-labeling tasks. Although large language models can generate fluent text, their outputs may also contain factual inconsistencies or unintended changes if not carefully controlled. Objective: This study investigated whether persona-driven, document-level DA using a large language model could improve biomedical disease NER performance by generating diverse rephrasings of medical documents while preserving annotated entities. Methods: We designed a DA framework using multiple personas that varied in medical expertise, personality, tone, and narrative style. Using prompting constrained by XML tags, each persona rephrased training documents while aiming to preserve annotated entity spans. We evaluated the framework on 2 biomedical disease NER datasets with complementary roles: RareDis, a low-resource rare disease corpus, and National Center for Biotechnology Information (NCBI) disease, a more general disease benchmark. Semantic fidelity and lexical diversity were measured using BERTScore and Bilingual Evaluation Understudy (BLEU-4), respectively, and personas were grouped into high-, balanced-, and low-fidelity subsets. Biomedical pretrained BioBERT models were fine-tuned and evaluated under multiple settings, including gold-standard (GS) data only, synonym replacement, single-persona augmentation, curated persona subsets, and all-persona augmentation. Performance was assessed using microaveraged entity-level precision, recall, and F1-score, and results were examined at both the overall and individual entity-type levels. Performance values are reported as mean (SD). Results: Persona-driven DA improved NER performance over GS-only training in both datasets, with the strongest gains obtained by combining multiple persona-generated variants with GS data. In RareDis, the best result was achieved by the low-fidelity subset (mean F1-score 73.35, SD 0.19 vs baseline 71.22, SD 0.45), while in NCBI disease, the all-personas setting performed best (mean F1-score 89.32, SD 0.26 vs baseline 87.82, SD 0.18). In low-resource experiments, the all-personas and high-fidelity persona settings in NCBI disease exceeded the performance of the model trained on 100% GS data using only 60% of the training data, whereas gains in RareDis were more modest. Entity-level analysis showed improvements across RareDis categories, particularly for symptom, and confusion analysis indicated reduced symptom-sign confusion under augmentation. Conclusions: Persona-driven DA improved biomedical disease NER by introducing controlled linguistic variation while largely preserving annotated entities. The strongest gains were obtained when multiple persona-generated variants were combined with GS data, although the benefit varied across datasets. These findings suggest that this approach is a promising strategy for low-resource biomedical NER.

  • Open Access

    A Multiassessment and Multiprofessional Agents Approach for Medical Chatbot Risk Estimation: Development and Evaluation Study (Preprint)

    by Lenard Paulo Velasco Tamayo, Tomohiro Nishiyama, Shaowen Peng, Shoko Wakamiya, Eiji Aramaki

    BACKGROUND Assessing chatbot responses across 3 domains—medical, ethical, and legal—is essential to ensuring the safe use of artificial intelligence in health care. Although advancements in the use of large language models (LLMs) show significant improvements in evaluating question-answer datasets, such as multiple-choice medical exams, existing systems use general LLMs without incorporating specialized domain knowledge. They rely on standardized instructions without integrating real-world information, and ensemble methods such as majority voting fail to resolve disagreements among agents, resulting in misclassification and challenges in risk assessment. OBJECTIVE This study aims to design, develop, and evaluate a synergistic approach for assessing risks associated with chatbot responses using multiassessment (MA) and multiprofessional agents (MPAs). METHODS We designed and developed an approach consisting of MA and MPA, specifically initial assessment (MA1), which internalizes 3 roles and provides an initial risk estimation, and final assessment (MA3), which aims to reach a final consensus based on the previous assessments (MA1 and MA2), with each using 1 LLM. The verification assessment (MA2) incorporates an MPA or role-based LLM specialized agents for each risk domain (medical, ethical, and legal). We evaluated the proposed approach using the MedNLP-CHAT (Medical Natural Language Processing for AI Chat) corpus (N=226; 100 train, 126 test), covering baseline, enhanced prompt, embedding-based search, and retrieval-augmented generation (RAG). Primary metrics included macro F1-score and joint accuracy to evaluate system performance, along with CI and paired macro F1-score difference (Δ) as supporting metrics to assess the approach’s effectiveness. RESULTS The MA-MPA framework integrated with RAG achieved the highest average macro F1-score of 0.800 across risk domains and a joint accuracy of 76 (60.3%) correct predictions across all risk domains out of 126 question-answer pairs, with notable improvements over the best reported Eighteenth NII Testbeds and Community for Information Access Research Project (NTCIR-18) MedNLP-CHAT systems in the ethical (+0.252) and legal (+0.096) risk domains, while the medical domain showed a modest increase of +0.070. The MA approach contributed the largest gains, particularly from MA1 to MA2, with paired macro F1-score gains ranging from +0.176 to +0.214 across systems. The MPA approach performed better when integrated with MA and external knowledge, with paired bootstrap estimates showing a gain of +0.037 (95% CI 0.003-0.074) over baseline; however, joint accuracy gains were not evident (95% CI –2.9% to 7.7%), and gains relative to the enhanced prompt were small. Notably, MA alone achieved higher joint accuracy than RAG (62.7% vs 60.3%), indicating a metric-specific trade-off rather than consistent superiority across all metrics. CONCLUSIONS The MA-MPA approach shows potential for improving risk estimation in chatbot responses. The results suggest that the framework is particularly useful for enhancing balanced overall performance, especially when combined with external knowledge, although the medical risk domain remains challenging. Furthermore, more specialized LLMs may further improve contextually grounded risk estimation.