Comparing performance between clinics of an embryo evaluation algorithm based on time-lapse images and machine learning
Open Access
- 10 July 2023
- journal article
- research article
- Published by Springer Nature in Journal of Assisted Reproduction and Genetics
- Vol. 40 (9) , 2129-2137
Abstract
Purpose: This article aims to assess how differences in maternal age distributions between IVF clinics affect the performance of an artificial intelligence model for embryo viability prediction and proposes a method to account for such differences. Methods: Using retrospectively collected data from 4805 fresh and frozen single blastocyst transfers of embryos incubated for 5 to 6 days, the discriminative performance was assessed based on fetal heartbeat outcomes. The data was collected from 4 clinics, and the discrimination was measured in terms of the area under ROC curves (AUC) for each clinic. To account for the different age distributions between clinics, a method for age-standardizing the AUCs was developed in which the clinic-specific AUCs were standardized using weights for each embryo according to the relative frequency of the maternal age in the relevant clinic compared to the age distribution in a common reference population. Results: There was substantial variation in the clinic-specific AUCs with estimates ranging from 0.58 to 0.69 before standardization. The age-standardization of the AUCs reduced the between-clinic variance by 16%. Most notably, three of the clinics had quite similar AUCs after standardization, while the last clinic had a markedly lower AUC both with and without standardization. Conclusion: The method of using age-standardization of the AUCs that is proposed in this article mitigates some of the variability between clinics. This enables a comparison of clinic-specific AUCs where the difference in age distributions is accounted for.Keywords
This publication has 32 references indexed in Scilit:
- Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and ElaborationAnnals of Internal Medicine, 2015
- Area under Precision-Recall Curves for Weighted and Unweighted DataPLOS ONE, 2014
- Assessing discriminative ability of risk models in clustered dataBMC Medical Research Methodology, 2014
- Assessing the Performance of Prediction ModelsEpidemiology, 2010
- Adjusting for covariate effects on classification accuracy using the covariate-adjusted receiver operating characteristic curveBiometrika, 2009
- Adjusting receiver operating characteristic curves and related indices for covariatesJournal of the Royal Statistical Society: Series D (The Statistician), 2003
- Quantifying heterogeneity in a meta‐analysisStatistics in Medicine, 2002
- Blastocyst score affects implantation and pregnancy outcome: towards a single blastocyst transferFertility and Sterility, 2000
- Statistical methods in cancer research. Volume II--The design and analysis of cohort studies.1987
- Meta-analysis in clinical trialsControlled Clinical Trials, 1986