Deviation of Zipf's and Heaps' Laws in Human Languages with Limited Dictionary Sizes
Authors:
Open Access
- 30 January 2013
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 3 (1) , srep01082
Abstract
Zipf's law on word frequency and Heaps' law on the growth of distinct words are observed in Indo-European language family, but it does not hold for languages like Chinese, Japanese and Korean. These languages consist of characters and are of very limited dictionary sizes. Extensive experiments show that: (i) The character frequency distribution follows a power law with exponent close to one, at which the corresponding Zipf's exponent diverges. Indeed, the character frequency decays exponentially in the Zipf's plot. (ii) The number of distinct characters grows with the text length in three stages: It grows linearly in the beginning, then turns to a logarithmical form and eventually saturates. A theoretical model for writing process is proposed, which embodies the rich-get-richer mechanism and the effects of limited dictionary size. Experiments, simulations and analytical solutions agree well with each other. This work refines the understanding about Zipf's and Heaps' laws in human language systems.Keywords
All Related Versions
This publication has 40 references indexed in Scilit:
- Statistical Laws Governing Fluctuations in Word Use from Word Birth to Word DeathScientific Reports, 2012
- Culturomics meets random fractal theory: insights into long-range correlations of social and natural phenomena over the past two centuriesJournal of The Royal Society Interface, 2012
- Word statistics in Blogs and RSS feeds: Towards empirical universal evidenceJournal of Informetrics, 2007
- Quantifying the evolutionary dynamics of languageNature, 2007
- Modelling the dynamics of language deathNature, 2003
- The Faculty of Language: What Is It, Who Has It, and How Did It Evolve?Science, 2002
- Computational and evolutionary aspects of languageNature, 2002
- The small world of human languageProceedings Of The Royal Society B-Biological Sciences, 2001
- The evolution of languageProceedings of the National Academy of Sciences, 1999
- Markov Processes: Linguistics and Zipf's LawPhysical Review Letters, 1995