Language Modeling for Morphologically Rich Languages: Character-Aware Modeling for Word-Level Prediction
Authors:
Open Access
- 1 December 2018
- journal article
- Published by MIT Press in Transactions of the Association for Computational Linguistics
- Vol. 6, 451-465
Abstract
Neural architectures are prominent in the construction of language models (LMs). However, word-level prediction is typically agnostic of subword-level information (characters and character sequences) and operates over a closed vocabulary, consisting of a limited word set. Indeed, while subword-aware models boost performance across a variety of NLP tasks, previous work did not evaluate the ability of these models to assist next-word prediction in language modeling tasks. Such subword-level informed models should be particularly effective for morphologically-rich languages (MRLs) that exhibit high type-to-token ratios. In this work, we present a large-scale LM study on 50 typologically diverse languages covering a wide variety of morphological systems, and offer new LM benchmarks to the community, while considering subword-level information. The main technical contribution of our work is a novel method for injecting subword-level information into semantic word vectors, integrated into the neural language mo...Keywords
This publication has 3 references indexed in Scilit:
- From Feedforward to Recurrent LSTM Neural Networks for Language ModelingIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015
- A bit of progress in language modelingComputer Speech & Language, 2001
- Long Short-Term MemoryNeural Computation, 1997