Identification of the parallel documents from multilingual news websites
- 1 October 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
We present the initial results of our experiments on document alignment for the online news domain. Specifically, as apposed to cross-site comparable news alignment, we focus on the identification of parallel documents from within the same multilingual websites. In such a setting parallel news stories oftentimes turn out to be direct translations of each other with a tendency of sharing common media and displaying proximity in publication date. We leverage this domain-specific property of the data and propose a straightforward yet competitive heuristic that performs on par with a machine learning-based method in terms of precision, and outperforms a widely used bitext extraction system on a range of metrics. Moreover, this heuristic has allowed us to identify comparable documents overlooked by a human annotator. Although both rule- and learning-based methods that we present are language independent, we specifically focus on the Russian-Kazakh language pair as the present study is one of the initial steps towards a greater objective of building a corresponding parallel corpus and a machine translation system.Keywords
This publication has 14 references indexed in Scilit:
- On certain aspects of Kazakh part-of-speech tagging2014
- Predicting political preference of Twitter usersSocial Network Analysis and Mining, 2014
- Overviewing Important Aspects of the Last Twenty Years of Research in Comparable Corpora2013
- Predicting political preference of Twitter users2013
- Leveraging editor collaboration patterns in wikipedia2012
- A fast method for parallel document identification2007
- Improving Machine Translation Performance by Exploiting Non-Parallel CorporaComputational Linguistics, 2005
- Automatic Identification of Parallel Documents With Light or Without Linguistic Resources2005
- The Web as a Parallel CorpusComputational Linguistics, 2003
- Automatic construction of parallel English-Chinese corpus for cross-language information retrieval2000