PEPE: Scalable extraction of multi-modal protein language model representations
Preprint
Abstract
Motivation: Protein language models (PLMs) capture intricate amino-acid dependencies, producing embeddings that encode rich structural, functional, and evolutionary information. Despite their potential, current extraction workflows rely on arbitrary choices, with respect to embedding layer, pooling, and padding, that frequently yield suboptimal representations for feature extraction and downstream analyses. Large-scale embedding generation is further limited by inefficiencies in computation and memory: (i) accumulating all model outputs in memory before writing to disk causes severe bottlenecks, and (ii) repeatedly embedding identical sequences to extract different modes introduces redundant computation and drastically reduces throughput and scalability. Results: We introduce PEPE (Parallel Extraction for Protein Embeddings), a command-line tool and Python library that enables efficient, high-throughput, and multimodal extraction from protein language models. PEPE’s parallelized and streaming-based architecture achieves runtimes several orders of magnitude faster than sequential approaches. Unlike conventional methods—whose peak memory usage scales linearly with output size and fails when memory capacity is exceeded—PEPE maintains stable, low memory consumption, enabling multimodal embedding extraction even beyond available RAM. PEPE supports a wide range of state-of-the-art and custom PLMs through a simple, flexible interface. By combining scalability, robustness, and ease of use, PEPE allows researchers to generate massive, information-rich embedding datasets efficiently, and facilitate the discovery of optimal representations for structural, functional, and evolutionary downstream tasks. By streamlining the generation of diverse embedding configurations, PEPE provides researchers with the necessary data to identify high-performing latent states for specific biological contexts without requiring additional computational resources. Availability and Implementation: PEPE is a command-line tool written in Python and published under MIT license. The source code and documentation are available at https://github.com/csi-greifflab/pepe-cli . PEPE is also available for installation from PyPI under https://pypi.org/project/pepe-cli and deposited on Zenodo at https://zenodo.org/records/15912054 .Keywords
Funding Information
- Innovative Medicines Initiative (101007799)
This publication has 27 references indexed in Scilit:
- PLMFit: benchmarking transfer learning with protein language models for protein engineeringBriefings in Bioinformatics, 2025
- AntiFold: improved structure-based antibody design using inverse foldingBioinformatics Advances, 2024
- Do domain-specific protein language models outperform general models on immunology-related tasks?ImmunoInformatics, 2024
- Designing proteins with language modelsNature Biotechnology, 2024
- Interpretable feature extraction and dimensionality reduction in ESM2 for protein localization predictionBriefings in Bioinformatics, 2024
- Language model-based B cell receptor sequence embeddings can effectively encode receptor specificityNucleic Acids Research, 2023
- The immuneML ecosystem for machine learning analysis of adaptive immune receptor repertoiresNature Machine Intelligence, 2021
- Learned Embeddings from Deep Learning to Visualize and Predict Protein SetsCurrent Protocols, 2021
- Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequencesProceedings of the National Academy of Sciences, 2021
- Array programming with NumPyNature, 2020