What does domain-specific pretraining buy? A 170,000-parameter plant peptide language model against the ESM-2 scaling curve

Authors:
Preprint
Publisher website
Google Scholar
Abstract
Small signalling peptides (SSPs) coordinate development and stress response in plants, and the computational tools that identify them represent sequences using general-purpose protein language models trained across all known proteins. How much a representation learned on the SSP sequence space itself is worth, measured against generalist embeddings of varying size, has not to our knowledge been quantified. We pretrain PhytoPepBERT, a 169,567-parameter masked language model implemented from scratch in NumPy with no deep-learning frame-work, on 1,469 plant SSP precursors retrieved from UniProtKB spanning five peptide families (RALF, PSK, CLE, PEP, RGF) and a length-matched negative class of non-SSP plant proteins. Sequences are clustered at 70% alignment identity and every cross-validation fold is grouped by cluster, so no homologue of a held-out sequence appears in training. Domain pretraining is worth a great deal relative to where it starts. Family classification by k -nearest neighbours over frozen embeddings reaches 85.2% ± 2.4% against 58.0% ± 4.3% for an architecturally identical, identically initialised untrained backbone (chance 16.7%), and against 75.1% for dipeptide composition and 67.6% for BLOSUM62. The advantage is largest where the backbone stays frozen: linear probing gives 82.9% against 67.6% untrained, while full fine-tuning narrows the gap to 85.4% against 83.6%. Masked-residue accuracy is 20.5% against 3.4% untrained, elevated across all five peptide families and lowest on the non-SSP negatives. Benchmarked against ESM-2 at four scales under an identical protocol, those 169,567 domain-pretrained parameters buy approximately what 7.5M generalist parameters buy: PhytoPepBERT ties esm2_t6_8M_UR50D (86.1% ± 2.8%, two of five paired folds) and is overtaken thereafter, by 2.5 points at 35M, 7.0 at 150M and 11.6 at 650M, losing every paired fold above the smallest checkpoint. TAPE, the second representation the reference pipeline uses, leads by 3.6 points at 92M. Generalist scale therefore continues to pay on this task well beyond the point where a domain model of this size can follow. We report an exchange rate rather than a victory: domain pretraining delivers roughly a forty-fold parameter efficiency against the generalist curve, and nothing beyond it.
Funding Information
  • UMA (B1-2022-29)

This publication has 45 references indexed in Scilit: