Fangwen Zhou

Background: Deep learning, particularly encoder-only transformer architectures, has demonstrated excellent performance in biomedical literature classification, facilitating evidence-based medicine, and knowledge synthesis. However, the opacity of these models’ decision-making processes limits their clinical interpretability, trustworthiness, and widespread adoption. Traditional explainable artificial intelligence methods, such as Shapley Additive Explanations (SHAP) and integrated gradients (IG), address this issue but often incur substantial computational overhead for text classification. Generative large language models may offer a novel approach to generating interpretable, context-aware explanations as autonomous agents. Objective: As a proof-of-concept, the study aimed to investigate the effectiveness of GPT-4o as a standalone, end-to-end perturbation-based explainer for a BioLinkBERT text classifier. We compared its explanations against the SHAP partition explainer and IG as established baselines in terms of explanation faithfulness and semantic alignment. Methods: A stratified sample of 200 studies from the McMaster Premium Literature Service (PLUS) and Clinical Hedges databases was classified by a fine-tuned BioLinkBERT model for methodological rigor. The sampling specifically over-represented difficult, low-confidence predictions to rigorously test the explainers, with an equal number of studies sampled from each probability decile predicted by BioLinkBERT. GPT-4o, SHAP, and IG generated token-level feature attributions across a robust feature space of 80,901 tokens. GPT-based explanations were derived through a sophisticated, iterative masking perturbation workflow under 2 prompting schemes (token indices vs explicit subword tokens). Explanations were evaluated using a rank-based, modified area over the perturbation curve (AOPC), pairwise correlation analyses, and qualitative assessment of feature importance. Results: Among the 200 studies, 80,901 tokens were included, and feature attributions were generated by the 4 explainers (6369 unique tokens). SHAP (AOPC 0.222, 95% CI 0.200-0.244) and IG (AOPC 0.225, 95% CI 0.202-0.247) provided consistent explanations, effectively identifying tokens relevant to study rigor (eg, “randomized” and “blind”). In contrast, despite evaluating a larger perturbation space, the GPT-4o prompting schemes did not achieve comparable faithfulness (AOPC 0.025-0.029) and produced divergent token attributions. Correlation analysis demonstrated moderate alignment between SHAP and IG (Pearson r= 0.367), whereas GPT-4o exhibited limited correlation (Pearson r ≤0.032) with the established baselines. Sensitivity analyses isolating only correctly classified instances yielded similar trends. Additionally, the iterative application programming interface calls required for GPT made it significantly more computationally intensive and costly to execute, whereas IG was the most temporally efficient. Conclusions: Despite their advanced contextual capabilities, current generative large language models are limited when deployed as standalone perturbation explainers. The findings reveal that GPT-4o struggles to accurately synthesize mathematical feature importance through iterative masking, lacking the reliability of traditional explainable artificial intelligence frameworks. Future research could build upon this work and investigate specialized prompt engineering, whole-word recombination strategies, and hybrid frameworks.

Background: Myalgic encephalomyelitis (ME), also referred to as chronic fatigue syndrome (CFS), is a complex illness that typically presents with disabling fatigue and cognitive and functional impairment. The etiology and management of ME/CFS remain contentious and patients often describe their experiences through social media. Objective: We explored public discourse on Twitter (rebranded as X) to understand the concerns and priorities of individuals living with ME/CFS, with a focus on (1) the COVID-19 pandemic and (2) publication of the 2021 UK National Institute for Health and Care Excellence (NICE) guidelines on the diagnosis and management of ME/CFS. Methods: We used the Twitter application programming interface to collect tweets related to ME/CFS posted between January 1, 2010, and January 30, 2024. Tweets were sorted into 3 chronological periods (pre–COVID-19 pandemic, post–COVID-19 pandemic, and post-UK 2021 NICE Guidelines publication). A Robustly Optimized Bidirectional Embedding Representations from Transformers Pretraining Approach (RoBERTa) language processing model was used to categorize the sentiment of tweets as positive, negative, or neutral. We identified tweets that mentioned COVID-19, the UK NICE guidelines, and key themes identified through latent Dirichlet allocation (ie, fibromyalgia, research, and treatment). We sampled 1000 random tweets from each theme to identify subthemes and representative quotes. Results: We retrieved 906,404 tweets, of which 427,824 (47.2%) were neutral, 369,371 (40.75%) were negative, and 109,209 (12.05%) were positive. Over time, both the proportion of negative and positive tweets increased, and the proportion of neutral tweets decreased (P<.001 for all changes). Tweets mentioning fibromyalgia acknowledged similarities with ME/CFS, stigmatization associated with both disorders, and lack of effective treatments. Treatment-related tweets often described frustration with ME/CFS labeled as mental illness, dismissal of concerns by health care providers, and the need to seek out “good physicians” who viewed ME/CFS as a physical disorder. Tweets on research typically praised studies of biomarkers and biomedical therapies, called for greater investment in biomedical research, and expressed frustration with studies suggesting a biopsychosocial etiology for ME/CFS or supporting management with psychotherapy or graduated activity. Tweets about the UK NICE guidelines expressed frustration with the 2007 version that recommended cognitive behavioral therapy and graded exercise therapy, and a prolonged campaign by advocacy organizations to influence subsequent versions. Tweets showed high acceptance of the 2021 UK NICE guidelines, which were seen to validate ME/CFS as a biomedical disease and recommended against graded exercise therapy. Tweets about COVID-19 often noted overlaps between post–COVID-19 condition and ME/CFS, including claims of a common biological pathway, and advised there was no cure for either condition. Conclusions: Our findings suggest research is needed to inform how best to support patients’ engagement with evidence-based care. Furthermore, while patient involvement with ME/CFS research is critical, unmanaged intellectual conflicts of interest may threaten the trustworthiness of research efforts.

The exponential growth of digital information has led to an unprecedented expansion in the volume of unstructured text data. Efficient classification of these data is critical for timely evidence synthesis and informed decision-making in health care. Machine learning techniques have shown considerable promise for text classification tasks. However, multiclass classification of papers by study publication type has been largely overlooked compared to binary or multilabel classification. Addressing this gap could significantly enhance knowledge translation workflows and support systematic review processes. This study aimed to fine-tune and evaluate domain-specific transformer-based language models on a gold-standard dataset for multiclass classification of clinical literature into mutually exclusive categories: original studies, reviews, evidence-based guidelines, and nonexperimental studies. The titles and abstracts of McMaster's Premium Literature Service (PLUS) dataset comprising 162,380 papers were used for fine-tuning seven domain-specific transformers. Clinical experts classified the papers into four mutually exclusive publication types. PLUS data were split in an 80:10:10 ratio into training, validation, and testing sets, with the Clinical Hedges dataset used for external validation. A grid search evaluated the impact of class weight (CW) adjustments, learning rate (LR), batch size (BS), warmup ratio, and weight decay (WD), totaling 1890 configurations. Models were assessed using 10 metrics, including the area under the receiver operating characteristic curve (AUROC), the F₁-score (harmonic mean of precision and recall), and Matthew's correlation coefficient (MCC). The performance of individual classes was assessed using a one-to-rest approach, and overall performance was assessed using the macro average. Optimal models identified from validation results were further tested on both PLUS and Clinical Hedges, with calibration assessed visually. Ten best-performing models achieved macro AUROC≥0.99, F₁-score≥0.89, and MCC≥0.88 on the validation and testing sets. Performance declined on Clinical Hedges. Models were consistently better at classifying original studies and reviews. Biomedical Bidirectional Encoder Representations from Transformers (fine-tuned on biomedical text; BioBERT)-based models had superior calibration performance, especially for original studies and reviews. Optimal configurations for search included lower LRs (1 × 10-5 and 3 × 10-5), midrange BSs (32-128), and lower WD (0.005-0.010). CW adjustments improved recall but generally reduced performance on other metrics. Models generally struggled with accurately classifying nonexperimental and guideline studies, potentially due to class imbalance and content heterogeneity. This study used a comprehensive hyperparameter search to highlight the effectiveness of fine-tuned transformer models, notably BioBERT variants, for multiclass clinical literature classification. Although class weighting generally decreased overall performance, addressing class imbalance through alternative methods, such as hierarchical classification or targeted resampling, warrants future exploration. Hyperparameter configurations were crucial for robust performance, aligning with the previous literature. These findings support future modeling research and practical deployment in human-in-the-loop systems to support knowledge synthesis and translation workflows with the findings from this work.

BACKGROUND The exponential growth of digital information has led to an unprecedented expansion in the volume of unstructured text data. Efficient classification of these articles is critical for timely evidence synthesis and informed decision-making in healthcare. Machine learning techniques have shown considerable promise for text classification tasks. However, multiclass classification of articles by study publication type has been largely overlooked compared to binary or multilabel classification. Addressing this gap could significantlyThe objective of this study was to fine-tune and evaluate domain-specific transformer-based language models on a gold-standard dataset for multiclass classification of clinical literature into mutually exclusive categories: original study, review, evidence-based guideline, and non-experimental. enhance knowledge translation workflows and support systematic review processes. OBJECTIVE The objective of this study was to fine-tune and evaluate domain-specific transformer-based language models on a gold-standard dataset for multiclass classification of clinical literature into mutually exclusive categories: original study, review, evidence-based guideline, and non-experimental. METHODS The titles and abstracts of McMaster’s Premium LiteratUre Service (PLUS) dataset of 162,380 articles were used for fine-tuning 7 domain-specific transformers. Clinical experts classified articles into four mutually exclusive publication types. PLUS data were split 80:10:10 for training, validation, and testing, with Clinical Hedges used for external validation. A grid search evaluated the impact of class weight adjustments, learning rate, batch size, warmup ratio, and weight decay, totaling 1,890 configurations. Models were assessed using 10 metrics, including area under the receiver operating characteristic curve (AUROC), F1 score, and Matthew’s correlation coefficient (MCC). Performance of individual classes was assessed using a one-to-rest approach, and overall performance was assessed using macro average. Optimal models identified from validation results were further tested on both PLUS and Clinical Hedges datasets, with calibration assessed visually. RESULTS Ten best-performing models achieved macro AUROC ≥0.99, F1 ≥0.89 and MCC ≥0.88 on the validation and test sets. Performance declined on Clinical Hedges. Models were consistently better at classifying original studies and reviews. BioBERT-based models had superior calibration performance, especially for original studies and reviews. Optimal configurations for search included lower learning rates (1E-5 and 3E-5), mid-range batch sizes (32–128), and lower weight decay (0.005-0.010). Class weight adjustments improved recall but generally reduced performance in other metrics. Models generally struggled with accurately classifying non-experimental and guideline articles, potentially due to class imbalance and content heterogeneity. CONCLUSIONS This study utilized a comprehensive hyperparameter search to highlight the effectiveness of fine-tuned transformer models, notably BioBERT variants, for multiclass clinical literature classification. While class weighting generally decreased overall performance, addressing class imbalance through alternative methods such as hierarchical classification or targeted resampling warrants future exploration. Optimal hyperparameter configurations were crucial for robust performance, aligning with previous literature. These findings support future modelling research and the practical deployment in human-in-the-loop systems to support knowledge synthesis and translation workflows using optimal configurations found in this work.

BACKGROUND Myalgic encephalomyelitis (ME), also referred to as chronic fatigue syndrome (CFS), is a complex illness that typically presents with disabling fatigue and cognitive and functional impairment. The etiology and management of ME/CFS remain contentious and patients often describe their experiences through social media. OBJECTIVE We explored public discourse on Twitter (rebranded as X) to understand the concerns and priorities of individuals living with ME/CFS, with a focus on (1) the COVID-19 pandemic and (2) publication of the 2021 UK National Institute for Health and Care Excellence (NICE) guidelines on the diagnosis and management of ME/CFS. METHODS We used the Twitter application programming interface to collect tweets related to ME/CFS posted between January 1, 2010, and January 30, 2024. Tweets were sorted into 3 chronological periods (pre–COVID-19 pandemic, post–COVID-19 pandemic, and post-UK 2021 NICE Guidelines publication). A Robustly Optimized Bidirectional Embedding Representations from Transformers Pretraining Approach (RoBERTa) language processing model was used to categorize the sentiment of tweets as positive, negative, or neutral. We identified tweets that mentioned COVID-19, the UK NICE guidelines, and key themes identified through latent Dirichlet allocation (ie, fibromyalgia, research, and treatment). We sampled 1000 random tweets from each theme to identify subthemes and representative quotes. RESULTS We retrieved 906,404 tweets, of which 427,824 (47.2%) were neutral, 369,371 (40.75%) were negative, and 109,209 (12.05%) were positive. Over time, both the proportion of negative and positive tweets increased, and the proportion of neutral tweets decreased (P<.001 for all changes). Tweets mentioning fibromyalgia acknowledged similarities with ME/CFS, stigmatization associated with both disorders, and lack of effective treatments. Treatment-related tweets often described frustration with ME/CFS labeled as mental illness, dismissal of concerns by health care providers, and the need to seek out “good physicians” who viewed ME/CFS as a physical disorder. Tweets on research typically praised studies of biomarkers and biomedical therapies, called for greater investment in biomedical research, and expressed frustration with studies suggesting a biopsychosocial etiology for ME/CFS or supporting management with psychotherapy or graduated activity. Tweets about the UK NICE guidelines expressed frustration with the 2007 version that recommended cognitive behavioral therapy and graded exercise therapy, and a prolonged campaign by advocacy organizations to influence subsequent versions. Tweets showed high acceptance of the 2021 UK NICE guidelines, which were seen to validate ME/CFS as a biomedical disease and recommended against graded exercise therapy. Tweets about COVID-19 often noted overlaps between post–COVID-19 condition and ME/CFS, including claims of a common biological pathway, and advised there was no cure for either condition. CONCLUSIONS Our findings suggest research is needed to inform how best to support patients’ engagement with evidence-based care. Furthermore, while patient involvement with ME/CFS research is critical, unmanaged intellectual conflicts of interest may threaten the trustworthiness of research efforts.

This systematic review and meta‐analysis of randomised controlled trials (RCTs) aimed to evaluate the efficacy, safety, and tolerability of fluvoxamine for the outpatient management of COVID‐19. We conducted this review in accordance with the PRISMA 2020 guidelines. Literature searches were conducted in MEDLINE, EMBASE, International Pharmaceutical Abstracts, CINAHL, Web of Science, and CENTRAL up to 14 September 2023. Outcomes included incidence of hospitalisation, healthcare utilization (emergency room visits and/or hospitalisation), mortality, supplemental oxygen and mechanical ventilation requirements, serious adverse events (SAEs) and non‐adherence. Fluvoxamine 100 mg twice a day was associated with reductions in the risk of hospitalisation (risk ratio [RR] 0.75, 95% confidence interval [CI] 0.58–0.97; I ² = 0%) and reductions in the risk of healthcare utilization (RR 0.68, 95% CI 0.53–0.86; I ² = 0%). While no increased SAEs were observed, fluvoxamine 100 mg twice a day was associated with higher treatment non‐adherence compared to placebo (RR 1.61, 95% CI 1.22–2.14; I ² = 53%). In subgroup analyses, fluvoxamine reduced healthcare utilization in outpatients with BMI ≥30 kg/m², but not in those with lower BMIs. While fluvoxamine offers potential benefits in reducing healthcare utilization, its efficacy may be most pronounced in high‐risk patient populations. The observed non‐adherence rates highlight the need for better patient education and counselling. Future investigations should reassess trial endpoints to include outcomes relating to post‐COVID sequelaes. Registration: This review was prospectively registered on PROSPERO (CRD42023463829).