PLMFit: benchmarking transfer learning with protein language models for protein engineering
Open Access
- 2 July 2025
- journal article
- Published by Oxford University Press (OUP) in Briefings in Bioinformatics
- Vol. 26 (4)
Abstract
Protein language models (PLMs) have emerged as a useful resource for protein engineering applications. Transfer learning (TL) leverages pre-trained parameters to extract features to train machine learning models or adjust the weights of PLMs for novel tasks via fine-tuning (FT) through back-propagation. TL methods have shown potential for enhancing protein predictions performance when paired with PLMs, however there is a notable lack of comparative analyses that benchmark TL methods applied to state-of-the-art PLMs, identify optimal strategies for transferring knowledge and determine the most suitable approach for specific tasks. Here, we report PLMFit, a benchmarking study that combines, three state-of-the-art PLMs (ESM2, ProGen2, ProteinBert), with three TL methods (feature extraction, low-rank adaptation, bottleneck adapters) for five protein engineering datasets. We conducted over >3150 in silico experiments, altering PLM sizes and layers, TL hyperparameters and different training procedures. Our experiments reveal three key findings: (i) utilizing a partial fraction of PLM for TL does not detrimentally impact performance, (ii) the choice between feature extraction (FE) and fine-tuning is primarily dictated by the amount and diversity of data, and (iii) FT is most effective when generalization is necessary and only limited data is available. We provide PLMFit as an open-source software package, serving as a valuable resource for the scientific community to facilitate the FE and FT of PLMs for various applications.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Learning meaningful representations of protein sequencesNature Communications, 2022
- Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteinsCell Systems, 2022
- Informed training set design enables efficient machine learning-assisted directed protein evolutionCell Systems, 2021
- Optimization of therapeutic antibodies by predicting antigen specificity from antibody sequence via deep learningNature Biomedical Engineering, 2021
- Evaluating Protein Transfer Learning with TAPE.2019
- Deep generative models of genetic variation capture the effects of mutationsNature Methods, 2018
- MMseqs2 enables sensitive protein sequence searching for the analysis of massive data setsNature Biotechnology, 2017
- Deep mutational scanning: a new style of protein scienceNature Methods, 2014
- kClust: fast and sensitive clustering of large protein sequence databasesBMC Bioinformatics, 2013
- Prediction of Protein Secondary Structure at Better than 70% AccuracyJournal of Molecular Biology, 1993