Toward Reliable Synthetic Omics: Statistical Distances for Generative Models Evaluation
Preprint
Abstract
Background: Synthetic data generation is emerging as an approach to overcome the limitations of real-world data scarcity in omics studies, especially in precision medicine and oncology. Omics datasets, with their high dimensionality and relatively small sample sizes, often lead to overfitting, especially in deep learning models. Generative models offer a promising way to generate realistic synthetic data preserving the original data distribution. However, there is still no objective consensus on how to evaluate their performance. In this study, we set out to validate generative networks for transcriptomics data generation by using statistical distances as robust evaluation metrics. Results: We observe that statistical distances enable simultaneous evaluation of global and local data fidelity of generated synthetic data. Because these distances satisfy the properties of true metrics, they also enable formal hypothesis testing to assess whether generative models have in fact converged or are merely approaching the reference distribution. Crucially, optimizing for these distances was found to implicitly select models maximizing other widely used metrics of generative performance, providing evidence of their broad applicability. Overall, our findings indicate that the adoption of these metrics can play a key role in guiding the development of generative models across a wide range of domains.Keywords
Funding Information
- European Commission (HORIZON-EIC-2022-PATHFINDER-OPEN 01-01 programme (grant no. 101099066))
- NextGenerationEU - National Recovery and Resilience Plan (Research program CN00000013 “National Centre for HPC, Big Data and Quantum Computing”, Spoke 8: Insilico Medicine and Omics Data (CN 00000013 — Avviso n. 3138 del 16 dicembre 2021))
This publication has 27 references indexed in Scilit:
- Preserving privacy in healthcare: A systematic review of deep learning approaches for synthetic data generationComputer Methods and Programs in Biomedicine, 2024
- Contrastive Diffusion Model with Auxiliary Guidance for Coarse-to-Fine PET Reconstruction2023
- Synthetic data generation: State of the art in health care domainComputer Science Review, 2023
- Pros and cons of GAN evaluation measures: New developmentsComputer Vision and Image Understanding, 2022
- Generation and evaluation of privacy preserving synthetic health dataNeurocomputing, 2020
- Conditional Generative Adversarial Networks for Data Augmentation in Breast Cancer Classification2019
- RNA Sequencing of the Tumor Microenvironment in Precision Cancer ImmunotherapyTrends in Cancer, 2019
- GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classificationNeurocomputing, 2018
- A Next Generation Connectivity Map: L1000 Platform and the First 1,000,000 ProfilesCell, 2017
- Energy distanceWIREs Computational Statistics, 2015