A theoretical model for n-gram distribution in big data corpora
- 1 December 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 134-141
Abstract
There is a wide diversity of applications relying on the identification of the sequences of n consecutive words (n-grams) occurring in corpora. Many studies follow an empirical approach for determining the statistical distribution of the n-grams but are usually constrained by the corpora sizes, which for practical reasons stay far away from Big Data. However, Big Data sizes imply hidden behaviors to the applications, such as extraction of relevant information from Web scale sources. In this paper we propose a theoretical approach for estimating the number of distinct n-grams in each corpus. It is based on the Zipf-Mandelbrot Law and the Poisson distribution, and it allows an efficient estimation of the number of distinct 1-grams, 2-grams,..., 6-grams, for any corpus size. The proposed model was validated for English and French corpora. We illustrate a practical application of this approach to the extraction of relevant expressions from natural language corpora, and predict its asymptotic behaviour for increasingly large sizes.Keywords
This publication has 7 references indexed in Scilit:
- Zipf’s word frequency law in natural language: A critical review and future directionsPsychonomic Bulletin & Review, 2014
- Recursive n-gram hashing is pairwise independent, at bestComputer Speech & Language, 2010
- Mandelbrot's Model for Zipf's Law: Can Mandelbrot's Model Explain Zipf's Law for Language?Journal of Quantitative Linguistics, 2009
- Untangling Herdan's law and Heaps' law: Mathematical and informetric argumentsJournal of the American Society for Information Science and Technology, 2007
- A formal derivation of Heaps' LawInformation Sciences, 2005
- Block addressing indices for approximate text retrievalJournal of the American Society for Information Science, 2000
- A model for word clusteringJournal of the American Society for Information Science, 1992