Retrievability of Code Mixed Microblogs
Open Access
- 7 July 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 973-976
Abstract
Mixing multiple languages within the same document, a phenomenon called (linguistic) code mixing or code switching, is a frequent trend among multilingual users of social media. In the context of information retrieval (IR), code mixing may affect retrieval effectiveness due to the mixing of different vocabularies with different collection statistics within a single collection of documents. In this paper, we investigate the indexing and retrieval strategies for a mixed collection of documents, comprising of code-mixed and the monolingual documents. In particular, we address three alternative modes of indexing, namely (a) a single index for the two sub-collections; (b) a separate index for each sub-collection; and (c) a clustered index with two individual sub-collection statistics coupled with the overall one. We make use of the expected retrievability scores of the two classes of documents to empirically show that indexing strategies (a) and (b) mostly retrieve the monolingual documents at top ranks with standard retrieval approaches. Our experiments show that, by contrast, the clustered index (c) is able to alleviate this problem by improving the retrievability of the code-mixed documents.Keywords
Funding Information
- Science Foundation Ireland (13/RC/2106)
This publication has 6 references indexed in Scilit:
- Code Mixing: A Challenge for Language Identification in the Language of Social Media2014
- POS Tagging of English-Hindi Code-Mixed Social Media Content2014
- Retrievability2008
- An exploration of axiomatic approaches to information retrieval2005
- Cluster-based retrieval using language models2004
- Probabilistic models of information retrieval based on measuring the divergence from randomnessACM Transactions on Information Systems, 2002