Simple controls exceed best deep learning algorithms and reveal foundation model effectiveness for predicting genetic perturbations
Authors:
Preprint
Abstract
Modeling genetic perturbations and their effect on the transcriptome is a key area of pharmaceutical research. Due to the complexity of the transcriptome, there has been much excitement and development in deep learning (DL) because of its ability to model complex relationships. In particular, the transformer-based foundation model paradigm emerged as the gold-standard of predicting post-perturbation responses. However, understanding these increasingly complex models and evaluating their practical utility is lacking, along with simple but appropriate benchmarks to compare predictive methods. Here, we present a simple baseline method that outperforms both state of the art (SOTA) in DL and other proposed simpler neural architectures, setting a necessary benchmark to evaluate in the field of post-perturbation prediction. We also elucidate the utility of foundation models for the task of post-perturbation prediction via generalizable fine-tuning experiments that can be translated to different applications of transformer-based foundation models to tasks of interest. Furthermore, we provide a corrected version of a popular dataset used for benchmarking perturbation prediction models. Our hope is that this work will properly contextualize further development of DL models in the perturbation space with necessary control procedures.Keywords
All Related Versions
- Published version: Bioinformatics, 41 (6).
This publication has 39 references indexed in Scilit:
- A review of transformer models in drug discovery and beyondJournal of Pharmaceutical Analysis, 2024
- Large language model to multimodal large language model: A journey to shape the biological macromolecules to biological sciences and medicineMolecular Therapy Nucleic Acids, 2024
- scMulan: A Multitask Generative Pre-Trained Language Model for Single-Cell Analysis2024
- A mini-review on perturbation modelling across single-cell omic modalitiesComputational and Structural Biotechnology Journal, 2024
- Mapping information-rich genotype-phenotype landscapes with genome-scale Perturb-seqCell, 2022
- Machine learning for perturbational single-cell omicsCell Systems, 2021
- Single cells make big data: New challenges and opportunities in transcriptomicsCurrent Opinion in Systems Biology, 2017
- Perturb-Seq: Dissecting Molecular Circuits with Scalable Single-Cell RNA Profiling of Pooled Genetic ScreensCell, 2016
- A Multiplexed Single-Cell CRISPR Screening Platform Enables Systematic Dissection of the Unfolded Protein ResponseCell, 2016
- Genome-Scale CRISPR-Mediated Control of Gene Repression and ActivationCell, 2014