Open Access
- 14 February 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 36978-36994
Abstract
Recently, legal practice has seen a significant rise in the adoption of Artificial Intelligence (AI) for various core tasks. However, these technologies remain in their early stages and face challenges such as understanding complex legal reasoning, managing biased data, ensuring transparency, and avoiding misleading responses, commonly referred to as hallucinations. To address these limitations, this paper introduces Legal Query RAG (LQ-RAG), a novel Retrieval-Augmented Generation framework with a recursive feedback mechanism specifically designed to overcome the critical shortcomings of standard RAG implementations in legal applications. The proposed framework incorporates four key components: a custom evaluation agent, a specialized response generation model, a prompt engineering agent, and a fine-tuned legal embedding LLM. Together, these components effectively minimize hallucinations, improve domain-specific accuracy, and deliver precise, high-quality responses for complex queries. Experimental results demonstrate that the fine-tuned embedding LLM achieves a 13% improvement in Hit Rate and a 15% improvement in Mean Reciprocal Rank (MRR). Comparisons with general LLMs reveal a 24% performance gain when using the Hybrid Fine-Tuned Generative LLM (HFM), the specialized response generation model integrated into the LQ-RAG framework. Furthermore, LQ-RAG achieves a 23% improvement in relevance score over naive configurations and a 14% improvement over RAG with Fine-Tuned LLMs (FTM). These findings underscore the potential of domain-specific fine-tuned LLMs, combined with advanced RAG modules and feedback mechanisms, to significantly enhance the reliability and performance of AI in legal practice. The reliance of this study on a proprietary model as the evaluation agent, combined with the lack of feedback from human experts, highlights the need for improvement. Future efforts should focus on developing a specialized legal evaluation agent and enhancing its performance by incorporating feedback from domain experts.Keywords
Funding Information
- Institute of Information & Communications Technology Planning & Evaluation
- Korea Government (MSIT)
- IITP (Institute of Information & Communications Technology Planning & Evaluation)-ITRC
- Korea Government [Ministry of Science and Information and Communication Technology (ICT)] (IITP-2025-RS-2021-II211835)
This publication has 29 references indexed in Scilit:
- CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering2024
- Generation of Asset Administration Shell With Large Language Model Agents: Toward Semantic Interoperability in Digital Twins in the Context of Industry 4.0IEEE Access, 2024
- An analysis of large language models: their impact and potential applicationsKnowledge and Information Systems, 2024
- GPT-4 passes the bar examPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2024
- Almanac — Retrieval-Augmented Language Models for Clinical MedicineNEJM AI, 2024
- Enhancing Zero-Shot Crypto Sentiment With Fine-Tuned Language Model and Prompt EngineeringIEEE Access, 2024
- Large language models in law: A surveyAI Open, 2024
- Large Legal Fictions: Profiling Legal Hallucinations in Large Language ModelsJournal of Legal Analysis, 2024
- The imperative for regulatory oversight of large language models (or generative AI) in healthcarenpj Digital Medicine, 2023
- Learning similarity with cosine similarity ensembleInformation Sciences, 2015