An Empirical Model for n-gram Frequency Distribution in Large Corpora
- 6 May 2020
- book chapter
- Published by Springer Nature
- Vol. 12085, 840-851
Abstract
Statistical multiword extraction methods can benefit from the knowledge on the n-gram (\(n \ge 1\)) frequency distribution in natural language corpora, for indexing and time/space optimization purposes. The appearance of increasingly large corpora raises new challenges on the investigation of the large scale behavior of the n-gram frequency distributions, not typically emerging on small scale corpora. We propose an empirical model, based on the assumption of finite n-gram language vocabularies, to estimate the number of distinct n-grams in large corpora, as well as the sizes of the equal-frequency n-gram groups, which occur in the lower frequencies starting from 1. The model was validated for n-grams with \(1 \le n \le 6\), by a wide range of real corpora in English and French, from 60 million up to 8 billion words. These are full non-truncated corpora data, that is, their associated frequency data include the entire range of observed n-gram frequencies, from 1 up to the maximum. The model predicts the monotonic growth of the numbers of distinct n-grams until reaching asymptotic plateaux when the corpus size grows to infinity. It also predicts the non-monotonicity of the sizes of the equal-frequency n-gram groups as a function of the corpus size.
Keywords
This publication has 7 references indexed in Scilit:
- A theoretical model for n-gram distribution in big data corpora2016
- A Universal Rank-Size LawPLOS ONE, 2016
- Zipf’s word frequency law in natural language: A critical review and future directionsPsychonomic Bulletin & Review, 2014
- Deviation of Zipf's and Heaps' Laws in Human Languages with Limited Dictionary SizesScientific Reports, 2013
- A Brief History of Generative Models for Power Law and Lognormal DistributionsInternet Mathematics, 2004
- Multiword unit hybrid extraction2003
- On a Class of Skew Distribution FunctionsBiometrika, 1955