Improving genomic language model reliability under distribution shift
Preprint
Abstract
Transformer-based Genomic Language Models (GLMs) have achieved strong performance across diverse genomic prediction tasks. However, their tendency toward overconfident predictions—particularly on noisy or unfamiliar data—limits reliability. In genomics, where unknown species and novel variants are common, developing models robust to distribution shift is crucial for dependable predictions. Here, we analyze the impact of several common and novel uncertainty quantification (UQ) methods in the context of GLMs, evaluating their performance across diverse downstream genomic and metagenomic prediction tasks. Comparing model behavior on both in-distribution (ID) and out-of-distribution (OOD) data, we show that temperature scaling and epistemic neural networks are capable of improving classification reliability across multiple GLM architectures and domains. The software is available at: https://github.com/EESI/glm-epinet-pytKeywords
This publication has 24 references indexed in Scilit:
- Genomic language models: opportunities and challengesTrends in Genetics, 2025
- Large language models: a survey of their development, capabilities, and applicationsKnowledge and Information Systems, 2024
- Impact of database choice and confidence score on the performance of taxonomic classification using Kraken2aBIOTECH, 2024
- A review of uncertainty quantification in deep learning: Techniques, applications and challengesInformation Fusion, 2021
- Effective gene expression prediction from sequence by integrating long-range interactionsNature Methods, 2021
- Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methodsMachine Learning, 2021
- MMseqs2 enables sensitive protein sequence searching for the analysis of massive data setsNature Biotechnology, 2017
- Aleatory or epistemic? Does it matter?Structural Safety, 2009
- An introduction to ROC analysisPattern Recognition Letters, 2006
- Basic Local Alignment Search ToolJournal of Molecular Biology, 1990