Stemming resource-poor Indian languages
Open Access
- 3 October 2014
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Asian Language Information Processing
- Vol. 13 (3) , 1-26
Abstract
Stemming is a basic method for morphological normalization of natural language texts. In this study, we focus on the problem of stemming several resource-poor languages from Eastern India, viz., Assamese, Bengali, Bishnupriya Manipuri and Bodo. While Assamese, Bengali and Bishnupriya Manipuri are Indo-Aryan, Bodo is a Tibeto-Burman language. We design a rule-based approach to remove suffixes from words. To reduce over-stemming and under-stemming errors, we introduce a dictionary of frequent words. We observe that, for these languages a dominant amount of suffixes are single letters creating problems during suffix stripping. As a result, we introduce an HMM-based hybrid approach to classify the mis-matched last character. For each word, the stem is extracted by calculating the most probable path in four HMM states. At each step we measure the stemming accuracy for each language. We obtain 94% accuracy for Assamese and Bengali and 87%, and 82% for Bishnupriya Manipuri and Bodo, respectively, using the hybrid approach. We compare our work with Morfessor [Creutz and Lagus 2005]. As of now, there is no reported work on stemming for Bishnupriya Manipuri and Bodo. Our results on Assamese and Bengali show significant improvement over prior published work [Sarkar and Bandyopadhyay 2008; Sharma et al. 2002, 2003].Keywords
This publication has 30 references indexed in Scilit:
- An Improved Stemming Approach Using HMM for a Highly Inflectional Language2013
- A Fast Corpus-Based StemmerACM Transactions on Asian Language Information Processing, 2011
- Design and Development of a Stemmer for PunjabiInternational Journal of Computer Applications, 2010
- Indexing and stemming approaches for the Czech languageInformation Processing & Management, 2009
- Acquisition of Morphology of an Indic Language from Text CorpusACM Transactions on Asian Language Information Processing, 2008
- YASSACM Transactions on Information Systems, 2007
- Stemming in Agglutinative Languages: A Probabilistic Stemmer for TurkishLecture Notes in Computer Science, 2003
- Viewing morphology as an inference processArtificial Intelligence, 2000
- An evaluation of some conflation algorithms for information retrievalJournal of Information Science, 1981
- An algorithm for suffix strippingProgram: electronic library and information systems, 1980