SpanSeq: similarity-based sequence data splitting method for improved development and assessment of deep learning projects
Open Access
- 2 July 2024
- journal article
- research article
- Published by Oxford University Press (OUP) in NAR Genomics and Bioinformatics
- Vol. 6 (3) , lqae106
Abstract
The use of deep learning models in computational biology has increased massively in recent years, and it is expected to continue with the current advances in the fields such as Natural Language Processing. These models, although able to draw complex relations between input and target, are also inclined to learn noisy deviations from the pool of data used during their development. In order to assess their performance on unseen data (their capacity to generalize), it is common to split the available data randomly into development (train/validation) and test sets. This procedure, although standard, has been shown to produce dubious assessments of generalization due to the existing similarity between samples in the databases used. In this work, we present SpanSeq, a database partition method for machine learning that can scale to most biological sequences (genes, proteins and genomes) in order to avoid data leakage between sets. We also explore the effect of not restraining similarity between sets by reproducing the development of two state-of-the-art models on bioinformatics, not only confirming the consequences of randomly splitting databases on the model assessment, but expanding those repercussions to the model development. SpanSeq is available at https://github.com/genomicepidemiology/SpanSeq.Keywords
All Related Versions
Funding Information
- European Union (874735)
This publication has 51 references indexed in Scilit:
- Finding Protein and Nucleotide Similarities with FASTACurrent Protocols in Bioinformatics, 2016
- Prediction of subcellular locations of proteins: Where to proceed?Proteomics, 2010
- The Elements of Statistical Learning2009
- Assessment of PLSDA cross validationMetabolomics, 2008
- Prediction of protein subcellular localizationProteins: Structure, Function, and Bioinformatics, 2006
- Defining a similarity threshold for a functional protein sequence pattern: The signal peptide cleavage siteProteins: Structure, Function, and Bioinformatics, 1996
- The Nature of Statistical Learning Theory1995
- Selection of representative protein data setsProtein Science, 1992
- Database of homology‐derived protein structures and the structural meaning of sequence alignmentProteins: Structure, Function, and Bioinformatics, 1991
- Comparison of the predicted and observed secondary structure of T4 phage lysozymeBiochimica et Biophysica Acta (BBA) - Protein Structure, 1975