Critical assessment of missense variant effect predictors on disease-relevant variant data
Open Access
- 21 March 2025
- journal article
- research article
- Published by Springer Nature in Human Genetics
- Vol. 144 (2-3) , 281-293
Abstract
Regular, systematic, and independent assessments of computational tools that are used to predict the pathogenicity of missense variants are necessary to evaluate their clinical and research utility and guide future improvements. The Critical Assessment of Genome Interpretation (CAGI) conducts the ongoing Annotate-All-Missense (Missense Marathon) challenge, in which missense variant effect predictors (also called variant impact predictors) are evaluated on missense variants added to disease-relevant databases following the prediction submission deadline. Here we assess predictors submitted to the CAGI 6 Annotate-All-Missense challenge, predictors commonly used in clinical genetics, and recently developed deep learning methods. We examine performance across a range of settings relevant for clinical and research applications, focusing on different subsets of the evaluation data as well as high-specificity and high-sensitivity regimes. Our evaluations reveal notable advances in current methods relative to older, well-cited tools in the field. While meta-predictors tend to outperform their constituent individual predictors, several newer individual predictors perform comparably to commonly used meta-predictors. Predictor performance varies between high-specificity and high-sensitivity regimes, highlighting that different methods may be optimal for different use cases. We also characterize two potential sources of bias. Predictors that incorporate allele frequency as a predictive feature tend to have reduced performance when distinguishing pathogenic variants from very rare benign variants, and predictors trained on pathogenicity labels from curated variant databases often inherit gene-level label imbalances. Our findings help illuminate the clinical and research utility of modern missense variant effect predictors and identify potential areas for future development.Keywords
All Related Versions
Funding Information
- National Institutes of Health (U01HG012022, U24HG007346)
This publication has 75 references indexed in Scilit:
- The Human Gene Mutation Database (HGMD®): optimizing its use in a clinical diagnostic or research settingHuman Genetics, 2020
- VIPdb, a genetic Variant Impact Predictor DatabaseHuman Mutation, 2019
- UniProt genomic mapping for deciphering functional effects of missense variantsHuman Mutation, 2019
- Updated recommendation for the benign stand‐alone ACMG/AMP criterionHuman Mutation, 2018
- PERCH: A Unified Framework for Disease Gene PrioritizationHuman Mutation, 2016
- Protein function in precision medicine: deep understanding with machine learningFEBS Letters, 2016
- The Evaluation of Tools Used to Predict the Impact of Missense Variants Is Hindered by Two Types of CircularityHuman Mutation, 2015
- Predicting the Functional, Molecular, and Phenotypic Consequences of Amino Acid Substitutions using Hidden Markov ModelsHuman Mutation, 2012
- Functional annotations improve the predictive score of human disease-related mutations in proteinsHuman Mutation, 2009
- On the histogram as a density estimator:L 2 theoryProbability Theory and Related Fields, 1981