Defining Semantically Close Words of Kazakh Language with Distributed System Apache Spark
Open Access
- 27 September 2023
- journal article
- research article
- Published by MDPI AG in Big Data and Cognitive Computing
- Vol. 7 (4) , 160
Abstract
This work focuses on determining semantically close words and using semantic similarity in general in order to improve performance in information retrieval tasks. The semantic similarity of words is an important task with many applications from information retrieval to spell checking or even document clustering and classification. Although, in languages with rich linguistic resources, the methods and tools for this task are well established, some languages do not have such tools. The first step in our experiment is to represent the words in a collection in a vector form and then define the semantic similarity of the terms using a vector similarity method. In order to tame the complexity of the task, which relies on the number of word (and, consequently, of the vector) pairs that have to be combined in order to define the semantically closest word pairs, A distributed method that runs on Apache Spark is designed to reduce the calculation time by running comparison tasks in parallel. Three alternative implementations are proposed and tested using a list of target words and seeking the most semantically similar words from a lexicon for each one of them. In a second step, we employ pre-trained multilingual sentence transformers to capture the content semantics at a sentence level and a vector-based semantic index to accelerate the searches. The code is written in MapReduce, and the experiments and results show that the proposed methods can provide an interesting solution for finding similar words or texts in the Kazakh language.Keywords
This publication has 18 references indexed in Scilit:
- A principled methodology for comparing relatedness measures for clustering publicationsQuantitative Science Studies, 2020
- Distributed sentiment analysis of an agglutinative language via Spark by applying machine learning methods2019
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- A Supervised Multi-class Multi-label Word Embeddings Approach for Toxic Comment Classification2019
- A New Approach for Calculating Semantic Similarity between Words Using WordNet and Set TheoryProcedia Computer Science, 2019
- Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech2018
- MEANS: A medical question-answering system combining NLP techniques and semantic Web technologiesInformation Processing & Management, 2015
- Searching in one billion vectors: Re-rank with source coding2011
- Product Quantization for Nearest Neighbor SearchIEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
- From Frequency to Meaning: Vector Space Models of SemanticsJournal of Artificial Intelligence Research, 2010