More on the Zipf's Law and Gibrat's Law

-
Since last Wednesday I have been updating the paper on the universe of cities. I have further simplified the statistical contents as statistics is usually confusing. It is a good learning for me, to focus on the findings, and make statistics simpler. Knowing that less is more, the restraint from lecturing statistics is something to be learned.

It also occurred to me when rewriting the paper that the real findings are not that our data comply with the rank-size rule and proportionate growth mentioned in the last post, but rather, the data don't fit perfectly with these two regularities.

The well-established rank-size rule describing city population size has its limit. It cannot fit the whole distribution if truncated at the lower end, as shown by Jan Eeckhout (2004). It cannot fit very large cities if pool the universe of cities together instead of looking into each country separately, as shown by our study. The power-law function is a mathematically simple and elegant model but has its limitations. The second regularity of proportionate growth also depends on the subject under study. During the past several decades, it applies to the developed counties but not the whole universe.

In this figure, I plot the fitted power-law function in Zipf CDF format: log (city population size)  vs. log(rank of the city). There are data from three time periods: 1990, 2000 and 2010.
(The figure is weird to people studying statistics as the x-axis and y-axis are the opposite to a histogram, but this is how the size-rank rule is illustrated conventionally.)

We can see obviously from the figure that the power-law function underlying the rank-size rule does not fit very well when the city population size is very large, especially above the dotted line (1E6.5) in Figure 8. Although it is worth noticing that power-law distribution fits most of the cities, since only 117 cities (2.8%) in the 2010 universe have population sizes larger than 1E6.5, the function cannot fit the complete universe of cities.

Actually, it turns out the exponents also vary a lot in countries. And power-law exponents are very sensitive to the definition of city and sample size, as observed by Rosen and Resnick in an early paper in 1980. It would be very helpful to do literature review earlier. 

I also just realized these two laws are the two sides of the same coin. Gabaix (1999) shows that if cities grow randomly according to Gibrat’s Law, the limit distribution of city size converges to obey Zipf’ Law. 

Comments

Post a Comment

Popular posts from this blog

How to Draw Heatmap with Colorful Dendrogram

-
Image
Heatmap with Dendrogram: Color Branches and Labels This data visualization example include: * Hierarchical clustering, dendrogram, and heat map based on normalized odds ratios * The dendrogram was built separately to give color to dendrogram’s branches/labels based on clusters using  dendextend * Heatmap is made by  heatmap.2  from  gplots  using the built dendrogram * The rows are sorted by means from highest to lowest, it can be done in either the dendrogram or the  heatmap.2 * Use color palettes from  colorspace Data YRBSS (Youth Risk Behavior Surveillance System) survey data from CDC. This example plots the odds ratios of 63 behavioral questions on one question regarding disordered eating. The odds ratios are median centered by column and log2 transformed Version 1: Color both the branches and labels library (curl) # read file from google drive library (gplots) # heatmap.2 library (dendextend) # make and color ...
Read more

Power-law distribution (Pareto)& Zipf's Law: connection and how to fit the distribution of global city population

-
Image
- This post counts for 3 days. - In short,  We use the histogram to outline the distribution, use rank-frequency plot (log-log plot) to identify the power-law distribution, and use maximum likelihood estimation to obtain the parameters. -  I will show 1 st , how the power-law function CDF and PDF can be derived from Zipf CDF; and 2 nd , this least square estimation by excel is wrong and how to get the right one. Historical background on the power-law distribution. - Power-law distribution is as common as normal distribution in the real world. In the 19 th century, Italian economist Pareto realized that 20% of the population owns 80% the wealth, which is the famous 80/20 rule. He later named the power-law distribution as Pareto distribution, which is what we studied in the probability course. - In the 20 th century, another main contributor, Zipf made a similar discovery in linguistics about the frequency of words used. - Besides social wealth, many things fol...
Read more

eXtreme Gradient Boosting (XGBoost): Better than random forest or gradient boosting

-
Image
Ensemble Methods 1. Random Forest 2. Gradient boosting 3. Extreme Gradient Boosting Compared to Lasso and Best-subset Lasso Best-subset Compare MSEs Code I recently had the great pleasure to meet with Professor Allan Just and he introduced me to eXtreme Gradient Boosting (XGBoost). XGBoost is “one of the most loved machine learning algorithms at Kaggle”, it somehow combines the advantages of random forest and boosting. We will see later its predictive ability is indeed very strong. In the earlier post  Model Selection using Lasso and Best Subset  I mainly focused on model selection by best-subset, stepwise, or lasso. This post will focus on ensemble method: random forest, Gradient boosting, and eXtreme Gradient Boosting (XGBoost). An ensemble method is a technique that combines the predictions from multiple machine learning algorithms together to make more accurate predictions than any individual model. 1. Random Forest Use library  ran...
Read more