Systematic contextual biases in SegmentNT potentially relevant to other nucleotide transformer models
Open Access
- 17 July 2026
- journal article
- Published by Oxford University Press (OUP) in Nucleic Acids Research
- Vol. 54 (14)
Abstract
Recent advances in large language models have extended to genomic applications, yet model robustness relative to context is unclear. Here, we demonstrate two intrinsic biases (input sequence length and nucleotide position) affecting SegmentNT results, a model included with the Nucleotide Transformer that provides nucleotide-level predictions of biological features. We demonstrate that nucleotide position within the input sequence (beginning, middle, or end) alters the nature of SegmentNT’s raw prediction probabilities, which can be standardized to improve prediction consistency. While longer input sequence length improves model performance, diminishing returns suggest a surprisingly small input length of ∼3072 nucleotides might be sufficient for many applications. We further identify a 24-nucleotide periodic oscillation in SegmentNT’s prediction probabilities, revealing an intrinsic bias potentially linked to the model’s training tokenization (6-mers) and architecture. We identify potential approaches to account for these biases and provide generalizable insights for utilizing nucleotide-resolution functional prediction models.Keywords
Funding Information
- BrightFocus Foundation (A2020118F)
- BrightFocus Foundation (A2020161S)
- National Institutes of Health (1P30AG072946-01)
- University of Kentucky Alzheimer’s Disease Research Center (AG06833)
- University of Kentucky Alzheimer’s Disease Research Center (R35GM138636)
- University of Kentucky Alzheimer’s Disease Research Center (R01AG08273)
- University of Kentucky Alzheimer’s Disease Research Center (RF1AG082339)
- Alzheimer's Association (2019-AARG-644082)
- Predoctoral Drug Discovery Fellowship to Heberle
This publication has 57 references indexed in Scilit:
- Scaling up self-supervised learning for improved surgical foundation modelsMedical Image Analysis, 2025
- Rotary Position Embedding for Vision Transformer2024
- Accurate structure prediction of biomolecular interactions with AlphaFold 3Nature, 2024
- Foundation Models for Natural Language Processing2023
- Transcriptome variation in human tissues revealed by long-read sequencingNature, 2022
- Full-length transcript sequencing of human and mouse cerebral cortex identifies widespread isoform diversity and alternative splicingCell Reports, 2021
- The most popular genes in the human genomeNature, 2017
- Predicting effects of noncoding variants with deep learning–based sequence modelNature Methods, 2015
- Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learningNature Biotechnology, 2015
- APOE is a major susceptibility gene for Alzheimer's diseaseCurrent Opinion in Biotechnology, 1994