Contrasting Sequence with Structure: Pre-training Graph Representations with PLMs
Preprint
Abstract
Understanding protein function is vital for drug discovery, disease diagnosis, and protein engineering. While Protein Language Models (PLMs) pre-trained on vast protein sequence datasets have achieved remarkable success, equivalent Protein Structure Models (PSMs) remain underrepresented. We attribute this to the relative lack of high-confidence structural data and suitable pre-training objectives. In this context, we introduce BioCLIP, a contrastive learning framework that pre-trains PSMs by leveraging PLMs, generating meaningful per-residue and per-chain structural representations. When evaluated on tasks such as protein-protein interaction, Gene Ontology annotation, and Enzyme Commission number prediction, BioCLIP-trained PSMs consistently outperform models trained from scratch and further enhance performance when merged with sequence embeddings. Notably, BioCLIP approaches, or exceeds, specialized methods across all benchmarks using its singular pre-trained design. Our work addresses the challenges of obtaining quality structural data and designing self-supervised objectives, setting the stage for more comprehensive models of protein function. Source code is publicly available 2 .Keywords
This publication has 47 references indexed in Scilit:
- Cryo-Electron Microscopy Methodology: Current Aspects and Future DirectionsTrends in Biochemical Sciences, 2019
- Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequencesbioRxiv, 2019
- ElemCor: accurate data analysis and enrichment calculation for high-resolution LC-MS stable isotope labeling experimentsBMC Bioinformatics, 2019
- A High Efficient Biological Language Model for Predicting Protein–Protein InteractionsCells, 2019
- Clustering huge protein sequence sets in linear timeNature Communications, 2018
- SchNet – A deep learning architecture for molecules and materialsThe Journal of Chemical Physics, 2018
- UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searchesBioinformatics, 2014
- Large-Scale Prediction of Human Protein−Protein Interactions from Amino Acid Sequence Based on Latent Topic FeaturesJournal of Proteome Research, 2010
- Predicting protein function from sequence and structureNature Reviews Molecular Cell Biology, 2007
- The Protein Data BankNucleic Acids Research, 2000