LLMs Are Also Effective Embedding Models: An In-depth Overview
Open Access
- 23 September 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 44 (7) , 1-41
Abstract
Large language models (LLMs) have revolutionized natural language processing by achieving state-of-the-art performance across various tasks. Recently, their effectiveness as embedding models has gained attention, marking a paradigm shift from traditional encoder-only models like ELMo and BERT to decoder-only, large-scale LLMs such as GPT, LLaMA, and Mistral. This survey provides an in-depth overview of this transition, beginning with foundational techniques before the LLM era, followed by LLM-based embedding models through two main strategies to derive embeddings from LLMs. (1) Direct prompting: We mainly discuss the prompt designs and the underlying rationale for deriving competitive embeddings. (2) Data-centric tuning: We cover extensive aspects that affect tuning an embedding model, including model architecture, training objectives, data construction, and so on. Upon the above, we also cover advanced methods for producing embeddings from longer texts, multilingual, code, cross-modal data, as well as reasoning-aware and other domain-specific scenarios. Furthermore, we discuss factors affecting choices of embedding models, such as performance comparisons, dense vs. sparse embeddings, pooling strategies, and scaling laws. Lastly, the survey highlights the limitations and challenges in adapting LLMs for embeddings, including cross-task embedding quality, tradeoffs between efficiency and accuracy, low-resource, long-context, data bias, robustness, and so on. This survey serves as a valuable resource for researchers and practitioners by synthesizing current advancements, highlighting key challenges, and offering a comprehensive framework for future work aimed at enhancing the effectiveness and efficiency of LLMs as embedding models.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62572034, U22B2021, and U24B20143)
- State Key Laboratory of Complex & Critical Software Environment
- CIE–Tianyi Cloud Research Program
This publication has 56 references indexed in Scilit:
- Text clustering with large language model embeddingsInternational Journal of Cognitive Computing in Engineering, 2025
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open QuestionsACM Transactions on Information Systems, 2025
- Scaling Laws For Dense Retrieval2024
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval2024
- C-Pack: Packed Resources For General Chinese Embeddings2024
- ONCE: Boosting Content-based Recommendation with Both Open- and Closed-source Large Language Models2024
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- UnifieR: A Unified Retriever for Large-Scale Retrieval2023
- A Short Survey of Pre-trained Language Models for Conversational AI-A New Age in NLP2020
- code2vec: learning distributed representations of codeProceedings of the ACM on Programming Languages, 2019