UNLT: Urdu Natural Language Toolkit
Authors:
Open Access
- 19 January 2022
- journal article
- research article
- Published by Cambridge University Press (CUP) in Natural Language Engineering
- Vol. 29 (4) , 942-977
Abstract
This study describes a Natural Language Processing (NLP) toolkit, as the first contribution of a larger project, for an under-resourced language—Urdu. In previous studies, standard NLP toolkits have been developed for English and many other languages. There is also a dire need for standard text processing tools and methods for Urdu, despite it being widely spoken in different parts of the world with a large amount of digital text being readily available. This study presents the first version of theUNLT (Urdu Natural Language Toolkit)which contains three key text processing tools required for an Urdu NLP pipeline; word tokenizer, sentence tokenizer, and part-of-speech (POS) tagger. The UNLT word tokenizer employs a morpheme matching algorithm coupled with a state-of-the-art stochasticn-gram language model with back-off and smoothing characteristics for the space omission problem. The space insertion problem for compound words is tackled using a dictionary look-up technique. The UNLT sentence tokenizer is a combination of various machine learning, rule-based, regular-expressions, and dictionary look-up techniques. Finally, the UNLT POS taggers are based on Hidden Markov Model and Maximum Entropy-based stochastic techniques. In addition, we have developed large gold standard training and testing data sets to improve and evaluate the performance of new techniques for Urdu word tokenization, sentence tokenization, and POS tagging. For comparison purposes, we have compared the proposed approaches with several methods. Our proposed UNLT, the training and testing data sets, and supporting resources are all free and publicly available for academic use.Keywords
This publication has 37 references indexed in Scilit:
- Morpheme Matching Based Text Tokenization for a Scarce Resourced LanguagePLOS ONE, 2013
- Getting More Out of Biomedical Documents with GATE's Full Lifecycle Open Source Text AnalyticsPLOS Computational Biology, 2013
- An Information-Extraction System for Urdu---A Resource-Poor LanguageACM Transactions on Asian Language Information Processing, 2010
- Automatic Part of Speech Tagging for Arabic: An Experiment Using Bigram Hidden Markov Model2010
- Hidden Markov Model Based Part of Speech Tagger for UrduInformation Technology Journal, 2007
- UIMA: an architectural approach to unstructured information processing in the corporate research environmentNatural Language Engineering, 2004
- An empirical study of smoothing techniques for language modelingComputer Speech & Language, 1999
- Support vector machinesIEEE Intelligent Systems and their Applications, 1998
- A tutorial on hidden Markov models and selected applications in speech recognitionProceedings of the IEEE, 1989
- Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.Psychological Bulletin, 1968