GENA-LM: A Family of Open-Source Foundational DNA Language Models for Long Sequences
Preprint
Abstract
Recent advancements in genomics, propelled by artificial intelligence, have unlocked unprecedented capabilities in interpreting genomic sequences, mitigating the need for exhaustive experimental analysis of complex, intertwined molecular processes inherent in DNA function. A significant challenge, however, resides in accurately decoding genomic sequences, which inherently involves comprehending rich contextual information dispersed across thousands of nucleotides. To address this need, we introduce GENA-LM, a suite of transformer-based foundational DNA language models capable of handling input lengths up to 36,000 base pairs. Notably, integration of the newly-developed Recurrent Memory mechanism allows these models to process even larger DNA segments. We provide pre-trained versions of GENA-LM, demonstrating their capability for fine-tuning and addressing a spectrum of complex biological tasks with modest computational demands. While language models have already achieved significant breakthroughs in protein biology, GENA-LM showcases a similarly promising potential for reshaping the landscape of genomics and multi-omics data analysis. All models are publicly available on GitHub https://github.com/AIRI-Institute/GENA LM and HuggingFace https://huggingface.co/AIRI-Institute.Keywords
All Related Versions
- Published version: Nucleic Acids Research, 53 (2).
This publication has 41 references indexed in Scilit:
- Interpreting Chromosomal Rearrangements in the Context of 3-Dimentional Genome Organization: A Practical Guide for Medical GeneticsBiochemistry (Moscow), 2018
- Deep Contextualized Word Representations2018
- HOCOMOCO: towards a complete collection of transcription factor binding models for human and mouse via large-scale ChIP-Seq analysisNucleic Acids Research, 2017
- Neural Machine Translation of Rare Words with Subword Units2016
- Predicting effects of noncoding variants with deep learning–based sequence modelNature Methods, 2015
- Machine learning applications in genetics and genomicsNature Reviews Genetics, 2015
- ChIP‐exo Method for Identifying Genomic Location of DNA‐Binding Proteins with Near‐Single‐Nucleotide AccuracyCurrent Protocols in Molecular Biology, 2012
- A Survey on Transfer LearningIEEE Transactions on Knowledge and Data Engineering, 2009
- AP-1: a double-edged sword in tumorigenesisNature Reviews Cancer, 2003
- Cooperative Interaction of GATA-2 and AP1 Regulates Transcription of the Endothelin-1 GeneMolecular and Cellular Biology, 1995