Evaluating Expert Specialization in Mixture-of-Experts Antibody Language Models
Preprint
Abstract
A bstract: Antibody language models (AbLMs) show an impressive aptitude for learning antibody features, but tend to struggle learning the highly diverse, non-templated regions of antibodies. Existing AbLMs use dense architectures, where all model parameters attend to each amino acid token. We hypothesized that the modular nature of antibodies could benefit from a sparse mixture-of-experts (MoE) architecture, allowing specific parameters (referred to as ‘experts’) to specialize in distinct antibody features. While MoE architectures are widely adopted and optimized in natural language processing domains, they are less common in biological modeling. To this end, we assess existing MoE routing strategies and find that token-choice routing strategies outperform expert-choice routing, presumably due to their specialization in CDRH3 residues. We further optimized the token-choice router for AbLMs, by minimizing the routing of padding tokens to enable pre-training with varying sequence lengths. Finally, we show that a large-scale baseline antibody language model with a Top-2 MoE architecture (BALM-MoE), trained on a mixture of unpaired and paired antibody sequences, outperforms its dense counterpart with the same number of active parameters.Keywords
Funding Information
- National Institute of Allergy and Infectious Diseases (P01-AI177683)
- National Institute of Allergy and Infectious Diseases (U19-AI135995)
- National Institute of Allergy and Infectious Diseases (R01- AI171438)
- National Institute of Allergy and Infectious Diseases (P30-AI036214)
- National Institute of Allergy and Infectious Diseases (UM1-AI144462)
- James B. Pendleton Charitable Trust
This publication has 26 references indexed in Scilit:
- UDA-seq: universal droplet microfluidics-based combinatorial indexing for massive-scale multimodal single-cell sequencingNature Methods, 2025
- Addressing the antibody germline bias and its effect on language models for improved antibody designBioinformatics, 2024
- Improving antibody language models with native pairingPatterns, 2024
- Deciphering the language of antibodies using self-supervised learningPatterns, 2022
- AbLang: an antibody language model for completing antibody sequencesBioinformatics Advances, 2022
- Observed Antibody Space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequencesProtein Science, 2021
- CoV-AbDab: the coronavirus antibody databaseBioinformatics, 2020
- Commonality despite exceptional diversity in the baseline human antibody repertoireNature, 2019
- NWChem: A comprehensive and scalable open-source solution for large scale molecular simulationsComputer Physics Communications, 2010
- Somatic generation of antibody diversityNature, 1983