Learning the Language of the Microbiome with Transformers
Preprint
Abstract
Self-supervised pretraining has become central to biological machine learning, yet microbiome data remains comparatively underexplored in terms of both modeling approaches and evaluation frameworks. To address this gap, we present Atlas, a pretraining dataset of over 539,000 microbiome datapoints from the MGnify database. Using Atlas, we train the Waypoint family of microbiome foundation models: a series of GPT-2 style causal language models ranging from 6M to 170M parameters. We also introduce Compass, a curated benchmark of eight predictive tasks spanning biome classification, drug-microbiome interactions, drug degradation, and infant gut development. Using this benchmark, we compare the performance of Waypoint models against classical baselines and the existing MGM foundation model. Our results show that pretraining leads to consistent and significant improvements in downstream task performance, that both dataset scale and tokenization strategy impact model quality, and that pretraining is essential for achieving favorable scaling behavior. Furthermore, pretrained transformer models begin to reliably outperform classical methods once training data exceeds roughly 10,000 examples - a threshold that is attainable for modern microbiome studies. Finally, we demonstrate that the Waypoint models achieve state-of-the-art performance among microbiome foundation models. Overall, our work highlights the importance of large-scale self-supervised pretraining in this domain and establishes Atlas, Compass, and the Waypoint models as valuable resources for the research community in this emerging field.Keywords
This publication has 18 references indexed in Scilit:
- Advancing regulatory variant effect prediction with AlphaGenomeNature, 2026
- MGM as a Large‐Scale Pretrained Foundation Model for Microbiome Analyses in Diverse ContextsAdvanced Science, 2026
- Genome modeling and design across all domains of life with Evo 2bioRxiv, 2025
- Nucleotide Transformer: building and evaluating robust foundation models for human genomicsNature Methods, 2024
- Nutrient competition predicts gut microbiome restructuring under drug perturbationsbioRxiv, 2024
- Training Compute-Optimal Protein Language ModelsbioRxiv, 2024
- AMP‐BERT: Prediction of antimicrobial peptide function based on a BERT modelProtein Science, 2022
- MGnify: the microbiome sequence data analysis resource in 2023Nucleic Acids Research, 2022
- Developmental trajectory of the healthy human gut microbiota during the first 5 years of lifeCell Host & Microbe, 2021
- DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genomeBioinformatics, 2021