Building Dictionaries for Low Resource Languages: Challenges of Unsupervised Learning
Authors:
- Diellza Nagavci MatiDMDiellza Nagavci Mati
- Mentor HamitiMHMentor Hamiti
- Arsim SusuriASArsim Susuri
- Besnik SelimiBSBesnik Selimi
- Jaumin AjdariJAJaumin Ajdari
Open Access
- 1 July 2021
- journal article
- research article
- Published by International Association for Educators and Researchers (IAER) in Annals of Emerging Technologies in Computing
- Vol. 5 (3) , 52-58
Abstract
The development of natural language processing resources for Albanian has grown steadily in recent years. This paper presents research conducted on unsupervised learning-the challenges associated with building a dictionary for the Albanian language and creating part-of-speech tagging models. The majority of languages have their own dictionary, but languages with low resources suffer from a lack of resources. It facilitates the sharing of information and services for users and whole communities through natural language processing. The experimentation corpora for the Albanian language includes 250K sentences from different disciplines, with a proposal for a part-of-speech tagging tag set that can adequately represent the underlying linguistic phenomena. Contributing to the development of Albanian is the purpose of this paper. The results of experiments with the Albanian language corpus revealed that its use of articles and pronouns resembles that of more high-resource languages. According to this study, the total expected frequency as a means for correctly tagging words has been proven effective for populating the Albanian language dictionary.Keywords
This publication has 16 references indexed in Scilit:
- Unsupervised Machine Learning based Documents Clustering in UrduICST Transactions on Scalable Information Systems, 2018
- An Experimental Evaluation of Algorithms for Opinion Mining in Multi-domain Corpus in Albanian2018
- Deep Learning for Natural Language Processing and Language Modelling2018
- Development and Evaluation of Word Embeddings for Morphologically Rich Languages2018
- A Thorough Experimental Evaluation of Algorithms for Opinion Mining in Albanian2018
- Performance Evaluation of Text Categorization Algorithms Using an Albanian Corpus2018
- Natural Language ProcessingInternational Journal of Computer Sciences and Engineering, 2018
- Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features2018
- Tagging Efficiency Analysis on Part of Speech Taggers2017
- Machine Learning Algorithms for Natural Language Semantics and Cognitive Computing2016