BioVERSE: Representation Alignment of Biomedical Modalities to LLMs for Multi-Modal Reasoning
Preprint
Abstract
Recent advances in large language models (LLMs) and biomedical foundation models (BioFMs) have achieved strong results in biological text reasoning, molecular modeling, and single-cell analysis, yet they remain siloed in disjoint embedding spaces, limiting cross-modal reasoning. We present BIOVERSE (Biomedical Vector Embedding Realignment for Semantic Engagement), a two-stage approach that adapts pretrained BioFMs as modality encoders and aligns them with LLMs through lightweight, modality-specific projection layers. The approach first aligns each modality to a shared LLM space through independently trained projections, allowing them to interoperate naturally, and then applies standard instruction tuning with multi-modal data to bring them together for downstream reasoning. By unifying raw biomedical data with knowledge embedded in LLMs, the approach enables zero-shot annotation, cross-modal question answering, and interactive, explainable dialogue. Across tasks spanning cell-type annotation, molecular description, and protein function reasoning, compact BIOVERSE configurations surpass larger LLM baselines while enabling richer, generative outputs than existing BioFMs, establishing a foundation for principled multi-modal biomedical reasoning.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Evaluating the Utilities of Foundation Models in Single‐Cell Data AnalysisAdvanced Science, 2026
- CELLama: Foundation Model for Single Cell and Spatial Transcriptomics by Cell Embedding Leveraging Language Model AbilitiesAdvanced Science, 2025
- 85 million cells — and counting — at your fingertipsNature, 2024
- scGPT: toward building a foundation model for single-cell multi-omics using generative AINature Methods, 2024
- Evolutionary-scale prediction of atomic-level protein structure with a language modelScience, 2023
- BioT5: Enriching Cross-modal Integration in Biology with Chemical Knowledge and Natural Language Associations2023
- ProteinBERT: a universal deep-learning model of protein sequence and functionBioinformatics, 2022
- Highly accurate protein structure prediction with AlphaFoldNature, 2021
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- cisTopic: cis-regulatory topic modeling on single-cell ATAC-seq dataNature Methods, 2019