Sheng-Chieh Lu

Background: The rise in artificial intelligence tools, especially those competent at language interpretation and translation, enables opportunities to enhance patient-centered care. One might be the ability to rapidly and inexpensively create accurate translations of English language patient-reported outcome measures (PROMs) to facilitate global uptake. Currently, it is unclear if machine translation (MT) tools can produce sufficient translation quality for this purpose. Methodology: We used Generative Pretrained Transformer (GPT)-4, GPT-3.5, and Google Translate to translate the English versions of selected scales from the Breast-Q and Face-Q, two widely used PROMs assessing outcomes following breast and face reconstructive surgery, respectively. We used MT to forward and back translate the scales from English into Arabic, Vietnamese, Italian, Hungarian, Malay, and Dutch. We compared translation quality using the Metrics for Evaluation of Translation with Explicit Ordering (METEOR). We compared the scores between different translation versions using the Kruskal-Wallis test or analysis of variance as appropriate. Results: In forward translations, the METEOR scores significantly varied depending on target languages for all MT tools (p < 0.001), with GPT-4 having the highest scores in most languages. We detected significantly different scores among translators for all languages (p < .05), except for Italian (p = 0.59). In backward translations, MTs (GPT-4: 0.81 ± 0.10; GPT-3.5: 0.78 ± 0.12; Google Translate: 0.80 ± 0.06) received higher or compatible scores to human translations (0.76 ± 0.11) for all languages. The differences in backward translation scores by different forward translators were significant for all languages (p < 0.01; except for Italian, p = 0.2). The scores between different languages were also significantly different for all translators (p < 0.001). Conclusions: Our findings suggest that large language models provide high-quality PROM translations to support human translations to reduce costs. However, substituting human translation with MT is not advisable at the current stage.

Background: Understanding sample representativeness is key to interpreting findings from epidemiological research and applying these findings to broader populations. Though techniques for assessing sample representativeness are available, they rely on access to raw data detailing the population of interest which are often not readily available and may not be suitable for comparing large datasets. In reality, population-based data are often only available in an aggregated format. In this study, we aimed to examine the capability of population stability index (PSI), a popular metric to assess data drift for artificial intelligence studies, in detecting sample differences using population-based data. Method: We obtained United States cancer statistics from the National Cancer Institute's Surveillance, Epidemiology, and End Results (SEER) database. We queried the SEER 17-registry research database to obtain cancer count data by age, sex, and cancer site groups from the rate sessions of the SEER*State incidence database for 2000 and 2015 – 2020. We then calculated PSI scores to estimate yearly data distribution shift from 2015 to 2020 for each variable. We compared the PSI results to the Chi-Square and Cramér's V tests for the same comparisons. Results: Scores for PSI comparing age, sex, and cancer site distribution between years ranged widely from 2.96 to less than 0.01. In line with our expectations, we found moderate to substantial differences in cancer population characteristics between 2000 and all other included years using PSI. Despite small effect sizes (Cramér's V 0.01 – 0.09), Chi-Square tests were significant for most comparisons, indicating likely type-I error caused by our large sample. Conclusions: Population stability index can be used to examine sample differences in healthcare studies where only binned data are available or where large datasets may reduce the reliability of other metrics. Inclusion of PSI in epidemiological research will give greater confidence that results are representative of the general population.

Background: The complex work of public health nurses (PHNs) specifically related to mental health assessment, intervention, and outcomes makes it difficult to quantify and evaluate the improvement in client outcomes attributable to their interventions. Objectives: We examined heterogeneity across parents of infants served by PHNs receiving different interventions, compared the ability of traditional propensity scoring methods versus energy-balancing weight (EBW) techniques to adjust for the complex and stark differences in baseline characteristics among those receiving different interventions, and evaluated the causal effects of the quantity and variety of PHN interventions on client health and social outcomes. Methods: This retrospective study of 4,109 clients used existing Omaha System data generated during the routine documentation of PHN home visit data. We estimated the effects of intervention by computing and comparing weighted averages of the outcomes within the different treatment groups using two weighting methods: (a) inverse probability of treatment (propensity score) weighting and (b) EBWs. Results: Clients served by PHNs differed in baseline characteristics with clients with more signs/symptoms. Both weighting methods reduced heterogeneity in the sample. EBWs were more effective than inverse probability of treatment weighting in adjusting for multifaceted confounding and resulted in close balance of 105 baseline characteristics. Weighting the sample changed outcome patterns, especially when using EBWs. Clients who received more PHN interventions and a wider variety of them had improved knowledge, behavior, and status outcomes with no plateau over time, whereas the unweighted sample showed plateaus in outcomes over the course of home-visiting services. Discussion: Causal analysis of PHN-generated data demonstrated PHN intervention effectiveness for clients with mental health signs/symptoms. EBWs are a promising tool for evaluating the true causal effect of PHN home-visiting interventions.

Background: Cancer patients often experience treatment-related symptoms which, if uncontrolled, may require emergency department admission. We developed models identifying breast or genitourinary cancer patients at the risk of attending emergency department (ED) within 30-days and demonstrated the development, validation, and proactive approach to in-production monitoring of an artificial intelligence-based predictive model during a 3-month simulated deployment at a cancer hospital in the United States.Methods: We used routinely-collected electronic health record data to develop our predictive models. We evaluated models including a variational autoencoderk-nearest neighbors algorithm (VAE-kNN) and model behaviors with a sample containing 84,138 observations from 28,369 patients. We assessed the model during a 77-day production period exposure to live data using a proactively monitoring process with predefined metrics.Results: Performance of the VAE-kNN algorithm is exceptional (Area under the receiver-operating characteristics, AUC = 0.80) and remains stable across demographic and disease groups over the production period (AUC 0.74–0.82). We can detect issues in data feeds using our monitoring process to create immediate insights into future model performance.Conclusions: Our algorithm demonstrates exceptional performance at predicting risk of 30-day ED visits. We confirm that model outputs are equitable and stable over time using a proactive monitoring approach.

PURPOSE: Clinical management of patients receiving immune checkpoint inhibitors (ICIs) could be informed using accurate predictive tools to identify patients at risk of short-term acute care utilization (ACU). We used routinely collected data to develop and assess machine learning (ML) algorithms to predict unplanned ACU within 90 days of ICI treatment initiation. METHODS: We used aggregated electronic health record data from 7,960 patients receiving ICI treatments to train and assess eight ML algorithms. We developed the models using pre–SARS-COV-19 COVID-19 data generated between January 2016 and February 2020. We validated our algorithms using data collected between March 2020 and June 2022 (peri–COVID-19 sample). We assessed performance using area under the receiver operating characteristic curves (AUROC), sensitivity, specificity, and calibration plots. We derived intuitive explanations of predictions using variable importance and Shapley additive explanation analyses. We assessed the marginal performance of ML models compared with that of univariate and multivariate logistic regression (LR) models. RESULTS: Most algorithms significantly outperformed the univariate and multivariate LR models. The extreme gradient boosting trees (XGBT) algorithm demonstrated the best overall performance (AUROC, 0.70; sensitivity, 0.53; specificity, 0.74) on the peri–COVID-19 sample. The algorithm performance was stable across both pre– and peri–COVID-19 samples, as well as ICI regimen and cancer groups. Type of ICI agents, oxygen saturation, diastolic blood pressure, albumin level, platelet count, immature granulocytes, absolute monocyte, chloride level, red cell distribution width, and alcohol intake were the top 10 key predictors used by the XGBT algorithm. CONCLUSION: Machine learning algorithms trained using routinely collected data outperformed traditional statistical models when predicting 90-day ACU. The XGBT algorithm has the potential to identify high–ACU risk patients and enable preventive interventions to avoid ACU.

Background: Mobile health (mHealth) apps offer new opportunities to deliver psychological treatments for mental illness in an accessible, private format. The results of several previous systematic reviews support the use of app-based mHealth interventions for anxiety and depression symptom management. However, it remains unclear how much or how long the minimum treatment “dose” is for an mHealth intervention to be effective. Just-in-time adaptive intervention (JITAI) has been introduced in the mHealth domain to facilitate behavior changes and is positioned to guide the design of mHealth interventions with enhanced adherence and effectiveness. Objective: Inspired by the JITAI framework, we conducted a systematic review and meta-analysis to evaluate the dose effectiveness of app-based mHealth interventions for anxiety and depression symptom reduction. Methods: We conducted a literature search on 7 databases (ie, Ovid MEDLINE, Embase, PsycInfo, Scopus, Cochrane Library (eg, CENTRAL), ScienceDirect, and ClinicalTrials, for publications from January 2012 to April 2020. We included randomized controlled trials (RCTs) evaluating app-based mHealth interventions for anxiety and depression. The study selection and data extraction process followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We estimated the pooled effect size using Hedge g and appraised study quality using the revised Cochrane risk-of-bias tool for RCTs. Results: We included 15 studies involving 2627 participants for 18 app-based mHealth interventions. Participants in the intervention groups showed a significant effect on anxiety (Hedge g=–.10, 95% CI –0.14 to –0.06, I2=0%) but not on depression (Hedge g=–.08, 95% CI –0.23 to 0.07, I2=4%). Interventions of at least 7 weeks’ duration had larger effect sizes on anxiety symptom reduction. Conclusions: There is inconclusive evidence for clinical use of app-based mHealth interventions for anxiety and depression at the current stage due to the small to nonsignificant effects of the interventions and study quality concerns. The recommended dose of mHealth interventions and the sustainability of intervention effectiveness remain unclear and require further investigation.

Background and Purpose: The objective of this study was to examine the content validity of the Omaha System to represent integrative healthcare (IH) interventions.Methods: A two-step classification procedure was used to validate Omaha System target terms that can represent IH interventions. Target terms were initially sorted based on evidence of use in IH interventions, including systematic reviews published in scientific journals and the Omaha System Guidelines website. Three Omaha System and integrative nursing content experts reviewed and validated target terms based on their definitions. Expert comments were reviewed and addressed, and final decisions were reached by consensus.Results: The content validity of Omaha System target terms was established for 49 of 75 (65.3%) target terms for IH interventions. These 49 targets were employed in 1145 of 1639 (69.9%) interventions in all Omaha System guidelines available online.Implications for Practice: A majority of Omaha System target terms may be used to represent IH interventions. Use of the Omaha System may facilitate efficient, structured, and thorough IH data collection to leverage informatics technology for supporting IH intervention clinical decisions, delivery, evaluation, and knowledge discovery.

Background: In the United States, national guidelines suggest that aggressive cancer care should be avoided in the final months of life. However, guideline compliance currently requires clinicians to make judgments based on their experience as to when a patient is nearing the end of their life. Machine learning (ML) algorithms may facilitate improved end-of-life care provision for patients with cancer by identifying patients at risk of short-term mortality. Objective: This study aims to summarize the evidence for applying ML in ≤1-year cancer mortality prediction to assist with the transition to end-of-life care for patients with cancer. Methods: We searched MEDLINE, Embase, Scopus, Web of Science, and IEEE to identify relevant articles. We included studies describing ML algorithms predicting ≤1-year mortality in patients of oncology. We used the prediction model risk of bias assessment tool to assess the quality of the included studies. Results: We included 15 articles involving 110,058 patients in the final synthesis. Of the 15 studies, 12 (80%) had a high or unclear risk of bias. The model performance was good: the area under the receiver operating characteristic curve ranged from 0.72 to 0.92. We identified common issues leading to biased models, including using a single performance metric, incomplete reporting of or inappropriate modeling practice, and small sample size. Conclusions: We found encouraging signs of ML performance in predicting short-term cancer mortality. Nevertheless, no included ML algorithms are suitable for clinical practice at the current stage because of the high risk of bias and uncertainty regarding real-world performance. Further research is needed to develop ML models using the modern standards of algorithm development and reporting.

PURPOSE: Despite their promises, digital innovations have scarcely translated to technologies used in routine clinical practice, making the identification of barriers to successful implementation a research priority. Low levels of transdisciplinary skills represent such a barrier but so far, this has not been evaluated and compared between information technology (IT) and health care specialists. In this study, we evaluated the level of digital health literacy among IT and health care specialists. MATERIALS AND METHODS: An anonymous questionnaire was distributed to staff at a breast cancer unit and an IT department of two German universities in December 2020. The survey questionnaire consisted of the previously validated eHealth Literacy Assessment Toolkit and additional questions with respect to age, profession, and career stage. Mann-Whitney or Wilcoxon rank-sum tests and two-sample chi-square tests were used for the analysis. RESULTS: The survey was completed by 113 individuals: 70 (61.9%) IT specialists and 43 (38.1%) health care specialists. Health care specialists scored significantly higher on the health-related scales and IT specialists scored significantly higher on the digitally related scales. No single participant identified themselves to have the highest level of literacy on all survey questions (n = 0 of 113; 0%). Only one person (n = 1 of 113; 0.9%) consistently reported a high or the highest level of literacy. CONCLUSION: Although IT and health care specialists showed great literacy in their respective disciplines, only few individuals combined both digital and health care literacy. Multidisciplinary teams and transdisciplinary curricula are crucial to bridge skill gaps between disciplines and to drive the implementation of digital health initiatives.