PepBERT: Lightweight language models for bioactive peptide representation
Preprint
Abstract
Protein language models (pLMs) have been widely adopted for various protein and peptide-related downstream tasks and demonstrated promising performance. However, short peptides are significantly underrepresented in commonly used pLM training datasets. For example, only 2.8% of sequences in the UniProt Reference Cluster (UniRef) contain fewer than 50 residues, which potentially limits the effectiveness of pLMs for peptide-specific applications. Here, we present PepBERT, a lightweight and efficient peptide language model specifically designed for encoding peptide sequences. Two versions of the model—PepBERT-large (4.9 million parameters) and PepBERT-small (1.86 million parameters)—were pretrained from scratch using four custom peptide datasets and evaluated on nine peptide-related downstream prediction tasks. Both PepBERT models achieved performance superior to or comparable to the benchmark model, ESM-2 with 7.5 million parameters, on 8 out of 9 datasets. Overall, PepBERT provides a compact yet effective solution for generating high-quality peptide representations for downstream applications. By enabling more accurate representation and prediction of bioactive peptides, PepBERT can accelerate the discovery of food-derived bioactive peptides with health-promoting properties, supporting the development of sustainable functional foods and value-added utilization of food processing by-products. The datasets, source codes, pretrained models, and tutorials for the usage of PepBERT are available at https://github.com/dzjxzyd/PepBERT .Keywords
All Related Versions
- Published version: Future Foods, 13.
Funding Information
- National Science Foundation (2419880)
- Cognitive and Neurobiological Approaches to Plasticity Center of Biomedical Research Excellence of the NIH (P20GM113109)
This publication has 19 references indexed in Scilit:
- pLM4CPPs: Protein Language Model-Based Predictor for Cell Penetrating PeptidesJournal of Chemical Information and Modeling, 2025
- PeptideBERT: A Language Model Based on Transformers for Peptide Property PredictionThe Journal of Physical Chemistry Letters, 2023
- Bioinformatics approaches to discovering food-derived bioactive peptides: Reviews and perspectivesTrAC Trends in Analytical Chemistry, 2023
- UniDL4BioPep: a universal deep learning architecture for binary classification in peptide bioactivityBriefings in Bioinformatics, 2023
- Identification of potent antimicrobial peptides via a machine-learning pipeline that mines the entire space of peptide sequencesNature Biomedical Engineering, 2023
- Leveraging molecular structure and bioactivity with chemical language models for de novo drug designNature Communications, 2023
- Review and perspective on bioactive peptides: A roadmap for research, development, and future opportunitiesJournal of Agriculture and Food Research, 2022
- ProteinBERT: a universal deep-learning model of protein sequence and functionBioinformatics, 2022
- NetSolP: predicting protein solubility in Escherichia coli using language modelsBioinformatics, 2021
- Self-referencing embedded strings (SELFIES): A 100% robust molecular string representationMachine Learning: Science and Technology, 2020