Protein language models learn evolutionary statistics of interacting sequence motifs
Top Cited Papers
Open Access
- 28 October 2024
- journal article
- Published by National Academy of Sciences in Proceedings of the National Academy of Sciences
- Vol. 121 (45)
Abstract
Protein language models (pLMs) have emerged as potent tools for predicting and designing protein structure and function, and the degree to which these models fundamentally understand the inherent biophysics of protein structure stands as an open question. Motivated by a finding that pLM-based structure predictors erroneously predict nonphysical structures for protein isoforms, we investigated the nature of sequence context needed for contact predictions in the pLM Evolutionary Scale Modeling (ESM-2). We demonstrate by use of a “categorical Jacobian” calculation that ESM-2 stores statistics of coevolving residues, analogously to simpler modeling approaches like Markov Random Fields and Multivariate Gaussian models. We further investigated how ESM-2 “stores” information needed to predict contacts by comparing sequence masking strategies, and found that providing local windows of sequence information allowed ESM-2 to best recover predicted contacts. This suggests that pLMs predict contacts by storing motifs of pairwise contacts. Our investigation highlights the limitations of current pLMs and underscores the importance of understanding the underlying mechanisms of these models.Keywords
All Related Versions
Funding Information
- Jane Coffin Childs (N/A)
- Howard Hughes Medical Institute (N/A)
- National Institutes of Health (DP5OD026389)
- National Foundation of Science (MCB2032259)
- Swiss National Science Foundation (205321_192371)
This publication has 33 references indexed in Scilit:
- Design of protein-binding proteins from the target structure aloneNature, 2022
- Efficient generative modeling of protein sequences using simple autoregressive modelsNature Communications, 2021
- Highly accurate protein structure prediction with AlphaFoldNature, 2021
- Design of proteins from smaller fragments—learning from evolutionCurrent Opinion in Structural Biology, 2014
- The impact of splicing on protein domain architectureCurrent Opinion in Structural Biology, 2013
- HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignmentNature Methods, 2011
- Learning generative models for protein fold familiesProteins: Structure, Function, and Bioinformatics, 2010
- Starch Catabolism by a Prominent Human Gut Symbiont Is Directed by the Recognition of Amylose HelicesStructure, 2008
- Alternative splicing and genome complexityNature Genetics, 2001
- Dictionary of protein secondary structure: Pattern recognition of hydrogen‐bonded and geometrical featuresBiopolymers, 1983