Khaled El Emam

BACKGROUND Primary care clinicians face clinical complexity alongside administrative burden from documentation, inbox work, prescriptions, referrals, and care coordination. Poor electronic health record usability can add work and is associated with dissatisfaction and burnout. Advances in artificial intelligence (AI) create an opportunity to relieve these pressures, but tools must be co-designed with clinicians, fit workflows, and be evaluated to avoid repeating the workload and usability problems of the electronic health record era. OBJECTIVE To identify current and near-term AI capabilities that health care AI experts viewed as relevant to primary care and to describe associated primary care AI project ideas suitable for early co-design and evaluation. METHODS We conducted a modified virtual nominal group technique (vNGT) with nine health care AI experts, including five physicians. The consultation was divided between two virtual meetings with non-overlapping participant groups. Participants generated and discussed current and emerging AI capabilities and associated primary care applications through silent idea generation, round-robin sharing, and facilitated discussion. Session outputs were consolidated after both meetings into capability and project-idea lists, followed by an individual post-session ranking survey. Participants ranked their top five project ideas and top five AI capabilities; ranks were converted to weighted points (rank 1=5 points; rank 5=1 point) and summed across respondents to generate descriptive weighted sum scores. RESULTS Participants generated two outputs: AI capabilities describing what systems would need to do and project ideas describing where they could be applied. Project ideas clustered around administrative relief (digital medical office assistant for forms, referrals, medication reviews, and scheduling; an electronic health record inbox manager), clinical encounter augmentation (ambient scribe plus micro-prompts), and pharmacy workflows (end-to-end prescribing support). Other domains addressed patient navigation and escalation, proactive identification of care gaps, and implementation readiness. Rankings were completed by 8 (projects) and 7 (capabilities) respondents. The digital medical office assistant received the highest project score (24), followed by the inbox manager (17); ambient scribe plus micro-prompts and the pharmacy agent tied (13 each). Retrieval-augmented generation (20) and clinical reasoning (19) received the highest capability scores, followed by multimodal understanding and generation and long-context handling (14 each), and function or tool calling (12). Discussion emphasized administrative overload, augmentation rather than clinical automation, workflow fit, clinician-led evaluation, interoperability, and reimbursement. CONCLUSIONS Expert participants identified current and emerging AI capabilities and a focused set of near-term project ideas for workflow-embedded primary care AI, with emphasis on administrative relief, encounter support, reliable reasoning, and grounded generation. These findings provide a starting point for clinician-led co-design and prospective evaluation in real-world primary care settings, but successful implementation will depend on interoperability, rigorous evaluation, and viable payment models.

Background: Synthetic data generation (SDG) has emerged as a promising solution to address data scarcity in health care, where privacy concerns, regulatory barriers, and the high cost of data acquisition limit access to real patient datasets. Machine learning models in this domain often operate in low-data regimes, with training set sizes as low as 20 and a median dataset size of around 600 records—conditions that hinder model generalization and increase the risks of overfitting and bias. SDG addresses these challenges by producing artificial samples that mimic real-world patient data, enabling robust and privacy-preserving model development. Objective: This study was a comprehensive assessment of SDG-augmented training across a wide array of models—both pretrained and non-pretrained—for outcome prediction in 13 health care datasets. For small datasets of sizes 50 and 350 records, we answer 3 key questions: (1) Do pretrained SDG models generate more effective augmentations than their non-pretrained counterparts for small datasets? (2) Is augmentation beneficial for both pretrained and non-pretrained classifiers for small datasets? (3) Among 3 state-of-the-art classification models, which offers the best predictive performance on small datasets? The workload that this study aimed to improve was binary classification. Methods: The 3 classifiers considered were light gradient boosting trees, large language models (LLMs) adapted to tabular data, and Tabular Prior-Data Fitted Network (TabPFN), a transformer-based method that has become the new state of the art in terms of tabular data classification. Each classifier was augmented through different SDG methods: current state-of-the-art techniques (Bayesian networks, conditional tabular generative adversarial networks, tabular variational autoencoders, and sequential trees) and the use of LLMs for tabular data generation. Results: Augmented TabPFN demonstrated superior performance, yielding significantly higher area under the curve and integrated calibration index scores compared to other classifiers. Post hoc analysis revealed that, for the dataset sizes examined, SDG and LLM models exhibited overfitting tendencies. Notably, simple dataset augmentation through sampling with replacement achieved performance comparable to that of SDG-based and LLM-based augmentation methods for TabPFN, suggesting that gains were primarily driven by increased sample size rather than SDG. Conclusions: Given its strong performance and minimal computational overhead, we recommend augmenting TabPFN through sampling with replacement as the optimal approach for small-data binary classification tasks. This method achieves performance comparable to that of more complex SDG techniques while offering substantial computational advantages.

Background: Prognostic information is essential for decision-making in breast cancer management. In recent years, trials and clinical practice have emphasized genomic prognostication tools, despite clinicopathological methods being more affordable and accessible. PREDICT v3 is one such tool with promising results across cohorts. Advances in machine learning (ML), transfer learning, and ensemble methods provide opportunities to enhance these approaches, especially where missing data and model assumptions differ across diverse populations. Objective: This study evaluates the potential to improve survival prognostication in breast cancer. More precisely, we compare de novo ML, transfer learning from the pretrained prognostication model PREDICT v3, and a stacked ensemble approach. Methods: Data from the MA.27 trial (NCT00066573) were used for model training, with external validation on data from the Tamoxifen Exemestane Adjuvant Multinational trial (NCT00279448 and NCT00032136) and a US Surveillance, Epidemiology, and End Results cohort. Transfer learning was applied by re-estimating the parameters of the pretrained prognostic tool PREDICT v3. De novo ML included random survival forests and extreme gradient boosting, and the ensemble was implemented using weighted linear stacking of model predictions. Internal and external validation was assessed in terms of the integrated calibration index and discrimination. Shapley Additive Explanations values were used to explain model predictions and decision-curve analysis to facilitate the interpretation of performance differences. Results: Transfer learning, de novo random survival forest, and the stacked ensemble improved calibration in MA.27 over the pretrained model (integrated calibration index reduced from 0.042 in PREDICT v3 to ≤0.007) while discrimination remained comparable (AUROC increased from 0.738 in PREDICT v3 to 0.744-0.799). In decision-curve analysis, these approaches demonstrated consistently positive net benefit across clinically relevant thresholds, while PREDICT v3 lost net benefit beyond 7.5% predicted risk. Invalid PREDICT v3 predictions were observed in 23.8% to 25.8% of MA.27 individuals due to missing information. In contrast, ML models and the stacked ensemble predicted survival despite missing data. Across all models, patient age, nodal status, pathological grading, and tumor size had the highest Shapley Additive Explanations values, indicating their importance for survival prognostication. External validation in the US Surveillance, Epidemiology, and End Results cohort confirmed the benefits of transfer learning, RSF, and ensemble in terms of calibration while maintaining discrimination at comparable levels. In contrast, generalizability was limited in the Tamoxifen Exemestane Adjuvant Multinational trial, a cohort with a substantially different distribution of clinicopathological characteristics. Conclusions: This study demonstrates that transfer learning, de novo RSF, and a stacked ensemble can improve prognostication compared with the pretrained PREDICT v3, particularly in the presence of missing or uncertain inputs. Transportability may be limited in cohorts with different clinicopathological profiles, requiring local validation before clinical deployment. Ultimately, better survival estimation can provide more meaningful guidance in breast cancer care. Trial Registration: ClinicalTrials.gov NCT00066573; https://clinicaltrials.gov/study/NCT00066573, NCT00279448; https://clinicaltrials.gov/study/NCT00279448, NCT00032136; https://clinicaltrials.gov/study/NCT00032136

BACKGROUND Prognostic information is essential for decision-making in breast cancer management. In recent years, trials and clinical practice have emphasized genomic prognostication tools, despite clinicopathological methods being more affordable and accessible. PREDICT v3 is one such tool with promising results across cohorts. Advances in machine learning (ML), transfer learning, and ensemble methods provide opportunities to enhance these approaches, especially where missing data and model assumptions differ across diverse populations. OBJECTIVE This study evaluates the potential to improve survival prognostication in breast cancer. More precisely, we compare de novo ML, transfer learning from the pretrained prognostication model PREDICT v3, and a stacked ensemble approach. METHODS Data from the MA.27 trial (NCT00066573) were used for model training, with external validation on data from the Tamoxifen Exemestane Adjuvant Multinational trial (NCT00279448 and NCT00032136) and a US Surveillance, Epidemiology, and End Results cohort. Transfer learning was applied by re-estimating the parameters of the pretrained prognostic tool PREDICT v3. De novo ML included random survival forests and extreme gradient boosting, and the ensemble was implemented using weighted linear stacking of model predictions. Internal and external validation was assessed in terms of the integrated calibration index and discrimination. Shapley Additive Explanations values were used to explain model predictions and decision-curve analysis to facilitate the interpretation of performance differences. RESULTS Transfer learning, de novo random survival forest, and the stacked ensemble improved calibration in MA.27 over the pretrained model (integrated calibration index reduced from 0.042 in PREDICT v3 to ≤0.007) while discrimination remained comparable (AUROC increased from 0.738 in PREDICT v3 to 0.744-0.799). In decision-curve analysis, these approaches demonstrated consistently positive net benefit across clinically relevant thresholds, while PREDICT v3 lost net benefit beyond 7.5% predicted risk. Invalid PREDICT v3 predictions were observed in 23.8% to 25.8% of MA.27 individuals due to missing information. In contrast, ML models and the stacked ensemble predicted survival despite missing data. Across all models, patient age, nodal status, pathological grading, and tumor size had the highest Shapley Additive Explanations values, indicating their importance for survival prognostication. External validation in the US Surveillance, Epidemiology, and End Results cohort confirmed the benefits of transfer learning, RSF, and ensemble in terms of calibration while maintaining discrimination at comparable levels. In contrast, generalizability was limited in the Tamoxifen Exemestane Adjuvant Multinational trial, a cohort with a substantially different distribution of clinicopathological characteristics. CONCLUSIONS This study demonstrates that transfer learning, de novo RSF, and a stacked ensemble can improve prognostication compared with the pretrained PREDICT v3, particularly in the presence of missing or uncertain inputs. Transportability may be limited in cohorts with different clinicopathological profiles, requiring local validation before clinical deployment. Ultimately, better survival estimation can provide more meaningful guidance in breast cancer care. CLINICALTRIAL ClinicalTrials.gov NCT00066573; https://clinicaltrials.gov/study/NCT00066573, NCT00279448; https://clinicaltrials.gov/study/NCT00279448, NCT00032136; https://clinicaltrials.gov/study/NCT00032136

Background: Small datasets are common in health research. However, the generalization performance of machine learning models is suboptimal when the training datasets are small. To address this, data augmentation is one solution and is often used for imaging and time series data, but there are no evaluations on its potential benefits for tabular health data. Augmentation increases sample size and is seen as a form of regularization that increases the diversity of small datasets, leading them to perform better on unseen data. Objectives: Evaluate data augmentation using generative models on tabular health data and assess the impact of diversity versus increasing the sample size. Methods: Using 13 large health datasets, we performed a simulation to evaluate the impact of data augmentation on the prediction performance (as measured by the ROC-AUC, the area under the receiver operating characteristic curve) on binary classification gradient boosted decision tree models. Four different synthetic data generation models were evaluated. We also built a generalized linear mixed effect model to assess the variable importance for model performance improvements from augmentation. We illustrate the proposed method on seven small real datasets as an application. A comparison of augmentation with resampling (which is a proxy for a larger dataset with minimal impact on diversity) was performed. Results: Augmentation improves prognostic performance for datasets that have higher cardinality categorical variables and lower baseline ROC-AUC. No specific generative model consistently outperformed the others. For the seven small application datasets, augmenting the existing data results in an increase in ROC-AUC between 4.31% (ROC-AUC from 0.71 to 0.75) and 43.23% (ROC-AUC from 0.51 to 0.73), with an average 15.55% relative improvement, demonstrating the nontrivial impact of augmentation on small datasets ( p = 0.0078). Augmentation ROC-AUC was higher than resampling only ROC-AUC ( p = 0.016). The diversity of augmented datasets was higher than the diversity of resampled datasets ( p = 0.046). Conclusions: This study demonstrates that data augmentation using generative models can have a marked benefit in terms of improved predictive performance for machine learning models on tabular health data, but only for datasets that meet baseline data complexity and predictive performance criteria. Our mixed effect model identified the most influential characteristics of the dataset and can help end-users have a more realistic expectation of the augmentation performance for a new dataset. Furthermore, augmentation performed better when having a smaller dataset, which is consistent with the argument that greater data diversity due to augmentation is beneficial. Clinical trial registration: Not applicable.

Synthetic tabular data generation (SDG) is increasingly important in healthcare research and innovation while preserving patients’ privacy. However, ethical concerns remain, primarily over residual privacy vulnerability and insufficient oversight. This review analyzes the only published SDG regulatory guidelines to date, from United Kingdom, Singapore, and South Korea. All emphasize privacy, acknowledging synthetic data is not inherently free from disclosure risks. Thresholds for sufficiently low risk are yet to be determined.

Objective: To compare the predictive performance of supervised machine learning models to logistic regression in identifying youth with eating disorders at risk of a complex clinical course based on clinical characteristics from the first treatment episode. Methods: Clinical data from 327 youth treated at any level of care at the Children's Hospital of Eastern Ontario Eating Disorders Program (2018–2024) were extracted. Complex clinical course outcome was defined as either readmission after discharge or a treatment trajectory deviating from the expected step‐down in intensity, including return to the same or escalation to a higher level of care. Thirty‐four intake and discharge variables from the first treatment episode were used to train seven machine learning models and logistic regression using repeated nested cross‐validation. Performance was assessed by AUC and brier scores. Models using intake‐only versus intake plus discharge data were compared. A parsimonious model using the top 10 predictors was also evaluated. Results: Random forest model with intake and discharge data achieved the best performance ( AUC = 0.723; Brier = 0.176) that was significantly superior to logistic regression. Models trained on intake‐only data showed poor discrimination ( AUCs < 0.6). Including discharge data improved model performance across all algorithms. The most important predictor was weight change throughout treatment. Random forest performance declined when restricted to the top 10 predictors. Discussion: Supervised machine learning demonstrates improved predictive performance for eating disorder disease course outcomes compared to traditional statistical methods, especially in higher‐dimensionality settings. These findings support future application of machine learning to complex biopsychosocial datasets to advance precision medicine initiatives in the eating disorder field and better understand the etiology of disease trajectory.

Objective: In medical research and education, generative artificial intelligence/machine learning (AI/ML) models to synthesize artificial medical data can enable the sharing of high-quality data while preserving the privacy of patients. Given that such data is often high-dimensional, a relevant consideration is whether to synthesize the entire dataset when only a task-relevant subset is needed. This study evaluates how the number of variables in training impacts fidelity, utility, and privacy of the synthetic data (SD). Material and Methods: We used 12 cross-sectional medical datasets, defined a downstream task with corresponding core variables, and derived 6354 variants by adding adjunct variables to the core. SD was generated using 7 different generative models and evaluated for fidelity, downstream utility, and privacy. Mixed-effect models were used to assess the effect of adjunct variables on the respective evaluation metric, accounting for the medical dataset as a random component. Results: Fidelity was unaffected by the number of adjunct variables in 5/7 SDG models. Similarly, downstream utility remained stable in 6/7 (predictive task) and 5/7 (inferential task) SDG models. Where significant effects were observed, they were minimal, resulting, for example, in a 0.05 decrease in Area under the Receiver Operating Characteristic curve (AUROC) when adding 120 variables. Privacy was not impacted by the number of adjunct variables. Discussion: Our findings show that fidelity, utility, and privacy are preserved when generating a more comprehensive medical dataset than the task-relevant subset. Conclusion: Our findings support a cost-effective, utility, and privacy-preserving way of implementing SDG into medical research and education.