xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein
Authors:
- Bo ChenBCBo Chen
- Xingyi ChengXCXingyi Cheng
- Pan LiPLPan Li
- Yangli-Ao GengYGYangli-Ao Geng
- Jing GongJGJing Gong
- Zhilei BeiZBZhilei Bei
- Xu TanXTXu Tan
Preprint
Abstract
Protein language models have shown remarkable success in learning biological information from protein sequences. However, most existing models are limited by either autoencoding or autoregressive pre-training objectives, which makes them struggle to handle protein understanding and generation tasks concurrently. We propose a unified protein language model, xTrimoPGLM, to address these two types of tasks simultaneously through an innovative pre-training framework. Our key technical contribution is an exploration of the compatibility and the potential for joint optimization of the two types of objectives, which has led to a strategy for training xTrimoPGLM at an unprecedented scale of 100 billion parameters and 1 trillion training tokens. Our extensive experiments reveal that 1) xTrimoPGLM significantly outperforms other advanced baselines in 18 protein understanding benchmarks across four categories. The model also facilitates an atomic-resolution view of protein structures, leading to an advanced 3D structural prediction model that surpasses existing language model-based tools. 2) xTrimoPGLM not only can generate de novo protein sequences following the principles of natural ones, but also can perform programmable generation after supervised fine-tuning (SFT) on curated sequences. These results highlight the substantial capability and versatility of xTrimoPGLM in understanding and generating protein sequences, contributing to the evolving landscape of foundation models in protein science. Trained weight for the xTrimoPGLM model, and downstream datasets are available at https://huggingface.co/proteinglm .Keywords
This publication has 91 references indexed in Scilit:
- DockQ: A Quality Measure for Protein-Protein Docking ModelsPLOS ONE, 2016
- ProFold: Protein Fold Classification with Additional Structural Features and a Novel Ensemble ClassifierBioMed Research International, 2016
- UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searchesBioinformatics, 2014
- PredPPCrys: Accurate Prediction of Sequence Cloning, Protein Production, Purification and Crystallization Propensity from Protein Sequences Using Multi-Step Heterogeneous Feature Fusion and SelectionPLOS ONE, 2014
- IMGT(R), the international ImMunoGeneTics information system(R)Nucleic Acids Research, 2008
- The Pairwise Energy Content Estimated from Amino Acid Composition Discriminates between Folded and Intrinsically Unstructured ProteinsJournal of Molecular Biology, 2005
- Scoring function for automated assessment of protein structure template qualityProteins: Structure, Function, and Bioinformatics, 2004
- UniProt: the Universal Protein knowledgebaseNucleic Acids Research, 2004
- ZDOCK: An initial‐stage protein‐docking algorithmProteins: Structure, Function, and Bioinformatics, 2003
- SCOP: a Structural Classification of Proteins databaseNucleic Acids Research, 2000