From Fact Drafts to Operational Systems: Semantic Search in Legal Decisions Using Fact Drafts
Open Access
- 10 December 2024
- journal article
- research article
- Published by MDPI AG in Big Data and Cognitive Computing
- Vol. 8 (12) , 185
Abstract
This research paper presents findings from an investigation in the semantic similarity search task within the legal domain, using a corpus of 1172 Hungarian court decisions. The study establishes the groundwork for an operational semantic similarity search system designed to identify cases with comparable facts using preliminary legal fact drafts. Evaluating such systems often poses significant challenges, given the need for thorough document checks, which can be costly and limit evaluation reusability. To address this, the study employs manually created fact drafts for legal cases, enabling reliable ranking of original cases within retrieved documents and quantitative comparison of various vectorization methods. The study compares twelve different text embedding solutions (the most recent became available just a few weeks before the manuscript was written) identifying Cohere’s embed-multilingual-v3.0, Beijing Academy of Artificial Intelligence’s bge-m3, Jina AI’s jina-embeddings-v3, OpenAI’s text-embedding-3-large, and Microsoft’s multilingual-e5-large models as top performers. To overcome the transformer-based models’ context window limitation, we investigated chunking, striding, and last chunk scaling techniques, with last chunk scaling significantly improving embedding quality. The results suggest that the effectiveness of striding varies based on token count. Notably, employing striding with 16 tokens yielded optimal results, representing 3.125% of the context window size for the best-performing models. Results also suggested that from the models having 8192 token long context window the bge-m3 model is superior compared to jina-embeddings-v3 and text-embedding-3-large models in capturing the relevant parts of a document if the text contains significant amount of noise. The validity of the approach was evaluated and confirmed by legal experts. These insights led to an operational semantic search system for a prominent legal content provider.Keywords
This publication has 29 references indexed in Scilit:
- NOWJ at COLIEE 2023: Multi-task and Ensemble Approaches in Legal Information ProcessingThe Review of Socionetwork Strategies, 2024
- An efficient long-text semantic retrieval approach via utilizing presentation learning on short-textComplex & Intelligent Systems, 2023
- Self Supervised Bert for Legal Text Classification2023
- Semantic matching based legal information retrieval system for COVID-19 pandemicArtificial Intelligence and Law, 2023
- BERT_LF: A Similar Case Retrieval Method Based on Legal FactsWireless Communications and Mobile Computing, 2022
- BERT: a sentiment analysis odysseyJournal of Marketing Analytics, 2021
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- Check node unit for LDPC decoders based on one‐hot data representation of messagesElectronics Letters, 2015
- Semantics-Preserving Bag-of-Words Models and ApplicationsIEEE Transactions on Image Processing, 2010
- Similarity between Euclidean and cosine angle distance for nearest neighbor queries2004