Zipf’s law holds for phrases, not words
Open Access
- 11 August 2015
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 5 (1) , 12209
Abstract
With Zipf’s law being originally and most famously observed for word frequency, it is surprisingly limited in its applicability to human language, holding over no more than three to four orders of magnitude before hitting a clear break in scaling. Here, building on the simple observation that phrases of one or more words comprise the most coherent units of meaning in language, we show empirically that Zipf’s law for phrases extends over as many as nine orders of rank magnitude. In doing so, we develop a principled and scalable statistical mechanical method of random text partitioning, which opens up a rich frontier of rigorous text analysis via a rank ordering of mixed length phrases.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Random Texts Do Not Exhibit the Real Zipf's Law-Like Rank DistributionPLOS ONE, 2010
- Empirical Tests of Zipf’s Law Mechanism in Open Source Linux DistributionPhysical Review Letters, 2008
- The Size, Scale, and Shape of CitiesScience, 2008
- Emergence of tempered preferential attachment from optimizationProceedings of the National Academy of Sciences, 2007
- Two Regimes in the Frequency of Words and the Origins of Complex Lexicons: Zipf’s Law Revisited∗Journal of Quantitative Linguistics, 2001
- Zipf Distribution of U.S. Firm SizesScience, 2001
- World Wide Web scaling exponent from Simon’s 1955 modelPhysical Review E, 2001
- Linear relations in biomechanics: the statistics of scaling functionsJournal of Zoology, 1985
- A general theory of bibliometric and other cumulative advantage processesJournal of the American Society for Information Science, 1976
- ON A CLASS OF SKEW DISTRIBUTION FUNCTIONSBiometrika, 1955