Lisa Pilgram

Background: Synthetic data generation (SDG) has emerged as a promising solution to address data scarcity in health care, where privacy concerns, regulatory barriers, and the high cost of data acquisition limit access to real patient datasets. Machine learning models in this domain often operate in low-data regimes, with training set sizes as low as 20 and a median dataset size of around 600 records—conditions that hinder model generalization and increase the risks of overfitting and bias. SDG addresses these challenges by producing artificial samples that mimic real-world patient data, enabling robust and privacy-preserving model development. Objective: This study was a comprehensive assessment of SDG-augmented training across a wide array of models—both pretrained and non-pretrained—for outcome prediction in 13 health care datasets. For small datasets of sizes 50 and 350 records, we answer 3 key questions: (1) Do pretrained SDG models generate more effective augmentations than their non-pretrained counterparts for small datasets? (2) Is augmentation beneficial for both pretrained and non-pretrained classifiers for small datasets? (3) Among 3 state-of-the-art classification models, which offers the best predictive performance on small datasets? The workload that this study aimed to improve was binary classification. Methods: The 3 classifiers considered were light gradient boosting trees, large language models (LLMs) adapted to tabular data, and Tabular Prior-Data Fitted Network (TabPFN), a transformer-based method that has become the new state of the art in terms of tabular data classification. Each classifier was augmented through different SDG methods: current state-of-the-art techniques (Bayesian networks, conditional tabular generative adversarial networks, tabular variational autoencoders, and sequential trees) and the use of LLMs for tabular data generation. Results: Augmented TabPFN demonstrated superior performance, yielding significantly higher area under the curve and integrated calibration index scores compared to other classifiers. Post hoc analysis revealed that, for the dataset sizes examined, SDG and LLM models exhibited overfitting tendencies. Notably, simple dataset augmentation through sampling with replacement achieved performance comparable to that of SDG-based and LLM-based augmentation methods for TabPFN, suggesting that gains were primarily driven by increased sample size rather than SDG. Conclusions: Given its strong performance and minimal computational overhead, we recommend augmenting TabPFN through sampling with replacement as the optimal approach for small-data binary classification tasks. This method achieves performance comparable to that of more complex SDG techniques while offering substantial computational advantages.

Background: Prognostic information is essential for decision-making in breast cancer management. In recent years, trials and clinical practice have emphasized genomic prognostication tools, despite clinicopathological methods being more affordable and accessible. PREDICT v3 is one such tool with promising results across cohorts. Advances in machine learning (ML), transfer learning, and ensemble methods provide opportunities to enhance these approaches, especially where missing data and model assumptions differ across diverse populations. Objective: This study evaluates the potential to improve survival prognostication in breast cancer. More precisely, we compare de novo ML, transfer learning from the pretrained prognostication model PREDICT v3, and a stacked ensemble approach. Methods: Data from the MA.27 trial (NCT00066573) were used for model training, with external validation on data from the Tamoxifen Exemestane Adjuvant Multinational trial (NCT00279448 and NCT00032136) and a US Surveillance, Epidemiology, and End Results cohort. Transfer learning was applied by re-estimating the parameters of the pretrained prognostic tool PREDICT v3. De novo ML included random survival forests and extreme gradient boosting, and the ensemble was implemented using weighted linear stacking of model predictions. Internal and external validation was assessed in terms of the integrated calibration index and discrimination. Shapley Additive Explanations values were used to explain model predictions and decision-curve analysis to facilitate the interpretation of performance differences. Results: Transfer learning, de novo random survival forest, and the stacked ensemble improved calibration in MA.27 over the pretrained model (integrated calibration index reduced from 0.042 in PREDICT v3 to ≤0.007) while discrimination remained comparable (AUROC increased from 0.738 in PREDICT v3 to 0.744-0.799). In decision-curve analysis, these approaches demonstrated consistently positive net benefit across clinically relevant thresholds, while PREDICT v3 lost net benefit beyond 7.5% predicted risk. Invalid PREDICT v3 predictions were observed in 23.8% to 25.8% of MA.27 individuals due to missing information. In contrast, ML models and the stacked ensemble predicted survival despite missing data. Across all models, patient age, nodal status, pathological grading, and tumor size had the highest Shapley Additive Explanations values, indicating their importance for survival prognostication. External validation in the US Surveillance, Epidemiology, and End Results cohort confirmed the benefits of transfer learning, RSF, and ensemble in terms of calibration while maintaining discrimination at comparable levels. In contrast, generalizability was limited in the Tamoxifen Exemestane Adjuvant Multinational trial, a cohort with a substantially different distribution of clinicopathological characteristics. Conclusions: This study demonstrates that transfer learning, de novo RSF, and a stacked ensemble can improve prognostication compared with the pretrained PREDICT v3, particularly in the presence of missing or uncertain inputs. Transportability may be limited in cohorts with different clinicopathological profiles, requiring local validation before clinical deployment. Ultimately, better survival estimation can provide more meaningful guidance in breast cancer care. Trial Registration: ClinicalTrials.gov NCT00066573; https://clinicaltrials.gov/study/NCT00066573, NCT00279448; https://clinicaltrials.gov/study/NCT00279448, NCT00032136; https://clinicaltrials.gov/study/NCT00032136

BACKGROUND Prognostic information is essential for decision-making in breast cancer management. In recent years, trials and clinical practice have emphasized genomic prognostication tools, despite clinicopathological methods being more affordable and accessible. PREDICT v3 is one such tool with promising results across cohorts. Advances in machine learning (ML), transfer learning, and ensemble methods provide opportunities to enhance these approaches, especially where missing data and model assumptions differ across diverse populations. OBJECTIVE This study evaluates the potential to improve survival prognostication in breast cancer. More precisely, we compare de novo ML, transfer learning from the pretrained prognostication model PREDICT v3, and a stacked ensemble approach. METHODS Data from the MA.27 trial (NCT00066573) were used for model training, with external validation on data from the Tamoxifen Exemestane Adjuvant Multinational trial (NCT00279448 and NCT00032136) and a US Surveillance, Epidemiology, and End Results cohort. Transfer learning was applied by re-estimating the parameters of the pretrained prognostic tool PREDICT v3. De novo ML included random survival forests and extreme gradient boosting, and the ensemble was implemented using weighted linear stacking of model predictions. Internal and external validation was assessed in terms of the integrated calibration index and discrimination. Shapley Additive Explanations values were used to explain model predictions and decision-curve analysis to facilitate the interpretation of performance differences. RESULTS Transfer learning, de novo random survival forest, and the stacked ensemble improved calibration in MA.27 over the pretrained model (integrated calibration index reduced from 0.042 in PREDICT v3 to ≤0.007) while discrimination remained comparable (AUROC increased from 0.738 in PREDICT v3 to 0.744-0.799). In decision-curve analysis, these approaches demonstrated consistently positive net benefit across clinically relevant thresholds, while PREDICT v3 lost net benefit beyond 7.5% predicted risk. Invalid PREDICT v3 predictions were observed in 23.8% to 25.8% of MA.27 individuals due to missing information. In contrast, ML models and the stacked ensemble predicted survival despite missing data. Across all models, patient age, nodal status, pathological grading, and tumor size had the highest Shapley Additive Explanations values, indicating their importance for survival prognostication. External validation in the US Surveillance, Epidemiology, and End Results cohort confirmed the benefits of transfer learning, RSF, and ensemble in terms of calibration while maintaining discrimination at comparable levels. In contrast, generalizability was limited in the Tamoxifen Exemestane Adjuvant Multinational trial, a cohort with a substantially different distribution of clinicopathological characteristics. CONCLUSIONS This study demonstrates that transfer learning, de novo RSF, and a stacked ensemble can improve prognostication compared with the pretrained PREDICT v3, particularly in the presence of missing or uncertain inputs. Transportability may be limited in cohorts with different clinicopathological profiles, requiring local validation before clinical deployment. Ultimately, better survival estimation can provide more meaningful guidance in breast cancer care. CLINICALTRIAL ClinicalTrials.gov NCT00066573; https://clinicaltrials.gov/study/NCT00066573, NCT00279448; https://clinicaltrials.gov/study/NCT00279448, NCT00032136; https://clinicaltrials.gov/study/NCT00032136

Objective: In medical research and education, generative artificial intelligence/machine learning (AI/ML) models to synthesize artificial medical data can enable the sharing of high-quality data while preserving the privacy of patients. Given that such data is often high-dimensional, a relevant consideration is whether to synthesize the entire dataset when only a task-relevant subset is needed. This study evaluates how the number of variables in training impacts fidelity, utility, and privacy of the synthetic data (SD). Material and Methods: We used 12 cross-sectional medical datasets, defined a downstream task with corresponding core variables, and derived 6354 variants by adding adjunct variables to the core. SD was generated using 7 different generative models and evaluated for fidelity, downstream utility, and privacy. Mixed-effect models were used to assess the effect of adjunct variables on the respective evaluation metric, accounting for the medical dataset as a random component. Results: Fidelity was unaffected by the number of adjunct variables in 5/7 SDG models. Similarly, downstream utility remained stable in 6/7 (predictive task) and 5/7 (inferential task) SDG models. Where significant effects were observed, they were minimal, resulting, for example, in a 0.05 decrease in Area under the Receiver Operating Characteristic curve (AUROC) when adding 120 variables. Privacy was not impacted by the number of adjunct variables. Discussion: Our findings show that fidelity, utility, and privacy are preserved when generating a more comprehensive medical dataset than the task-relevant subset. Conclusion: Our findings support a cost-effective, utility, and privacy-preserving way of implementing SDG into medical research and education.

Synthetic data generation (SDG) plays an increasingly important role as a research and innovation accelerator. While SDG can enable privacy-preserving data sharing, it also raises privacy concerns compounded by uncertainty how privacy law applies to SDG and the generated data itself. Such uncertainty can hinder positive applications of SDG and put individual privacy rights at risk. This study aims to understand how SDG and synthetic data are treated under Canadian federal privacy law, identifying regulatory gaps that extend beyond the Canadian context and proposing recommendations to address them. Our analysis shows that SDG is not explicitly addressed by the statute. While SDG arguably qualifies as a use of personal information, it is unclear whether consent is required for SDG. Further Fair Information Practices with respective obligations apply to SDG just as they do to any use of personal information. The generated data itself could fall outside the law’s scope since it is more likely to qualify as non-personal than traditionally de-identified data but the concept of identifiability under the statute remains ambiguous, particularly regarding inferences. An unclear definition of identifiability represents a relevant gap in privacy law that can harm the individual directly, through the exposure of personal information, or indirectly, by hindering the adoption of SDG and other beneficial privacy-enhancing technologies. A Code of Practice, anchored in legislation, could address such privacy concerns and ensure the proper application of SDG.

Background: Generative artificial intelligence (AI) for tabular synthetic data generation (SDG) has significant potential to accelerate health care research and innovation. A critical limitation of generative AI, however, is hallucinations. Although this has been commonly observed in text-generating models, it may also occur in tabular SDG. Objective: This study aims to investigate the magnitude of hallucinations in tabular synthetic data, whether their frequency increases with training data complexity, and the extent to which they impact the utility of synthetic data for downstream prognostic machine learning (ML) modeling tasks. Methods: On the basis of 12 large and high-dimensional real-world health care datasets, 6354 training datasets of different complexity were created by varying the subset of variables included in each dataset. Synthetic data were generated using 7 different SDG models. Hallucinations were defined as synthetic records that did not exist in the population, and the hallucination rate (HR) was the proportion of hallucinations in a synthetic dataset. Classification was the downstream prognostic modeling task, conducted via an ML approach (light gradient boosted machine) and an artificial neural network (multilayer perceptron). Mixed-effects models were fitted to examine the relationship between training data complexity and the HR and the HR and the predictive performance of AI and ML models when trained on the synthetic data. Results: The HR ranged from 0.3% to 100% (median 99.1%, IQR 98.5%-100.0%) and increased with training data complexity. However, in most SDG models, the HR did not affect AI and ML prognostic model performance. In the SDG models in which a significant association was detected, the estimated effect was very small, with a maximum decrease in the area under the receiver operating characteristic curve of –0.0002 (95% CI –0.0003 to –0.0002, P<.001) in light gradient boosting machine and –0.0001 (95% CI –0.0002 to –0.0001, P=.002) in multilayer perceptron. Conclusions: These findings suggest that while hallucinations may be very common in synthetic tabular health data, they do not necessarily impair its utility for prognostic modeling.

BACKGROUND Generative artificial intelligence (AI) for tabular synthetic data generation (SDG) has significant potential to accelerate health care research and innovation. A critical limitation of generative AI, however, is hallucinations. Although this has been commonly observed in text-generating models, it may also occur in tabular SDG. OBJECTIVE This study aims to investigate the magnitude of hallucinations in tabular synthetic data, whether their frequency increases with training data complexity, and the extent to which they impact the utility of synthetic data for downstream prognostic machine learning (ML) modeling tasks. METHODS On the basis of 12 large and high-dimensional real-world health care datasets, 6354 training datasets of different complexity were created by varying the subset of variables included in each dataset. Synthetic data were generated using 7 different SDG models. Hallucinations were defined as synthetic records that did not exist in the population, and the hallucination rate (HR) was the proportion of hallucinations in a synthetic dataset. Classification was the downstream prognostic modeling task, conducted via an ML approach (light gradient boosted machine) and an artificial neural network (multilayer perceptron). Mixed-effects models were fitted to examine the relationship between training data complexity and the HR and the HR and the predictive performance of AI and ML models when trained on the synthetic data. RESULTS The HR ranged from 0.3% to 100% (median 99.1%, IQR 98.5%-100.0%) and increased with training data complexity. However, in most SDG models, the HR did not affect AI and ML prognostic model performance. In the SDG models in which a significant association was detected, the estimated effect was very small, with a maximum decrease in the area under the receiver operating characteristic curve of –0.0002 (95% CI –0.0003 to –0.0002, P<.001) in light gradient boosting machine and –0.0001 (95% CI –0.0002 to –0.0001, P=.002) in multilayer perceptron. CONCLUSIONS These findings suggest that while hallucinations may be very common in synthetic tabular health data, they do not necessarily impair its utility for prognostic modeling.