Building Dictionaries for Low Resource Languages: Challenges of Unsupervised Learning

Authors:
  • DMDiellza Nagavci Mati
    Diellza Nagavci Mati
  • MHMentor Hamiti
    Mentor Hamiti
  • ASArsim Susuri
    Arsim Susuri
  • BSBesnik Selimi
    Besnik Selimi
  • JAJaumin Ajdari
    Jaumin Ajdari

Abstract
The development of natural language processing resources for Albanian has grown steadily in recent years. This paper presents research conducted on unsupervised learning-the challenges associated with building a dictionary for the Albanian language and creating part-of-speech tagging models. The majority of languages have their own dictionary, but languages with low resources suffer from a lack of resources. It facilitates the sharing of information and services for users and whole communities through natural language processing. The experimentation corpora for the Albanian language includes 250K sentences from different disciplines, with a proposal for a part-of-speech tagging tag set that can adequately represent the underlying linguistic phenomena. Contributing to the development of Albanian is the purpose of this paper. The results of experiments with the Albanian language corpus revealed that its use of articles and pronouns resembles that of more high-resource languages. According to this study, the total expected frequency as a means for correctly tagging words has been proven effective for populating the Albanian language dictionary.