A review on the applications of Transformer-based language models for nucleotide sequence analysis
Authors:
Open Access
- 1 January 2025
- journal article
- review article
- Published by American Association for the Advancement of Science (AAAS) in Computational and Structural Biotechnology Journal
- Vol. 27, 1244-1254
Abstract
Transformer-based language models are making an impact in the field of Natural Language Processing (NLP). As relevant parallels can be drawn between biological sequences and natural languages, the models used in NLP can be easily extended and adapted for applications in bioinformatics. This paper introduces the recent developments of Transformer-based models in the context of nucleotide sequences. We have reviewed and analysed a large number of application-based papers on this subject, giving evidence of the main characterizing features and to the different approaches that may be adopted to customize such powerful computational machines. Besides discussing what Transformers do and may do for the analysis of biological sequences, we also provide an overview of what Transformers are and why they work. We believe this review will help the scientific community in understanding the application of Transformer-based language models to nucleotide sequences, and that will motivate the readers to build on idea of Transformers as well as the discussed methodologies to tackle different problems in the field of bioinformatics.Keywords
All Related Versions
This publication has 77 references indexed in Scilit:
- SENet: A deep learning framework for discriminating super- and typical enhancers by sequence informationComputational Biology and Chemistry, 2023
- BERT-Promoter: An improved sequence-based predictor of DNA promoter using BERT pre-trained model and SHAP feature selectionComputational Biology and Chemistry, 2022
- CHEER: HierarCHical taxonomic classification for viral mEtagEnomic data via deep leaRningMethods, 2021
- Pre-trained models: Past, present and futureAI Open, 2021
- Representation learning applications in biological sequence analysisComputational and Structural Biotechnology Journal, 2021
- Identifying viruses from metagenomic data using deep learningQuantitative Biology, 2020
- An integrated expression atlas of miRNAs and their promoters in human and mouseNature Biotechnology, 2017
- Identifying 2′-O-methylationation sites by integrating nucleotide chemical properties and nucleotide compositionsGenomics, 2016
- Epigenetic polymorphism and the stochastic formation of differentially methylated regions in normal and cancerous tissuesNature Genetics, 2012
- Human DNA methylomes at base resolution show widespread epigenomic differencesNature, 2009