Mining the web to create minority language corpora
Authors:
Open Access
- 5 October 2001
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 279-286
Abstract
The Web is a valuable source of language specific resources but the process of collecting, organizing and utilizing these resources is difficult. We describe CorpusBuilder, an approach for automatically generating Web-search queries for collecting documents in a minority language. It differs from pseudo-relevance feedback in that retrieved documents are labeled by an automatic language classifier as relevant or irrelevant, and this feedback is used to generate new queries. We experiment with various query-generation methods and query-lengths to find inclusion/exclusion terms that are helpful for retrieving documents in the target language and find that using odds-ratio scores calculated over the documents acquired so far was one of the most consistently accurate query-generation methods. We also describe experiments using a handful of words elicited from a user instead of initial documents and show that the methods perform similarly. Experiments applying the same approach to multiple languages are also presented showing that our approach generalizes to a variety of languages.Keywords
This publication has 7 references indexed in Scilit:
- Learning a monolingual language model from a multilingual text database2000
- Document Categorization and Query Generation on the World Wide Web Using WebACEArtificial Intelligence Review, 1999
- Automatic discovery of language models for text databases1999
- A Winnow-Based Approach to Context-Sensitive Spelling CorrectionMachine Learning, 1999
- Mining the Web for bilingual text1999
- Relevance feedback and inference networks1993
- Relevance weighting of search termsJournal of the American Society for Information Science, 1976