A Review on the Applications of Transformer-based language models for Nucleotide Sequence Analysis
Authors:
Preprint
Abstract
In recent times, Transformer-based language models are making quite an impact in the field of natural language processing. As relevant parallels can be drawn between biological sequences and natural languages, the models used in NLP can be easily extended and adapted for various applications in bioinformatics. In this regard, this paper introduces the major developments of Transformer-based models in the recent past in the context of nucleotide sequences. We have reviewed and analysed a large number of application-based papers on this subject, giving evidence of the main characterizing features and to different approaches that may be adopted to customize such powerful computational machines. We have also provided a structured description of the functioning of Transformers, that may enable even first time users to grab the essence of such complex architectures. We believe this review will help the scientific community in understanding the various applications of Transformer-based language models to nucleotide sequences. This work will motivate the readers to build on these methodologies to tackle also various other problems in the field of bioinformatics.Keywords
All Related Versions
This publication has 74 references indexed in Scilit:
- SENet: A deep learning framework for discriminating super- and typical enhancers by sequence informationComputational Biology and Chemistry, 2023
- Improving language model of human genome for DNA–protein binding prediction based on task-specific pre-trainingInterdisciplinary Sciences: Computational Life Sciences, 2022
- BERT-Promoter: An improved sequence-based predictor of DNA promoter using BERT pre-trained model and SHAP feature selectionComputational Biology and Chemistry, 2022
- CHEER: HierarCHical taxonomic classification for viral mEtagEnomic data via deep leaRningMethods, 2021
- Pre-trained models: Past, present and futureAI Open, 2021
- Representation learning applications in biological sequence analysisComputational and Structural Biotechnology Journal, 2021
- Identifying viruses from metagenomic data using deep learningQuantitative Biology, 2020
- Identifying 2′-O-methylationation sites by integrating nucleotide chemical properties and nucleotide compositionsGenomics, 2016
- Topological maps of protein sequencesBiological Cybernetics, 1991
- THE AMINO ACID COMPOSITION OF ANIMAL TISSUE PROTEINJournal of Biological Chemistry, 1943