How I mapped the city-level GDP from MGI dataset into the universe of cities

-
One of the main exercises I did during the late year 2017 was to create city-level GDP for the 4231 "universe" of cities. There were lots of interesting problems encountered during the process.

The main reference dataset is McKinsey Global Institute (MGI)'s Cityscope database of 2910 cities. They have city GDP and population, and they highly correlate with each other. I guess regression was already applied in obtaining their city GDP data.

My own guideline on this task is to rely on MGI data as much as I can. The 1st step was to match cities directly.  I first matched by city name and country, then matched those within 20km geological distance.  Because not surprising, there are no common code or city names. Even the country name can be different and sometimes MGI's geological coordinations are wrong.

There are 2617 (62%) cities matched directly by MGI data. Then linear regressions within each country using MGI city population as the independent variable to fit, and universe population as the independent variable to predict, helps to get another 966 (23%) cities. Regression works well in countries with enough cities. For example, China alone takes up 24% of the universe. But notice that among the 172 countries in our universe, 65% (112) countries have fewer than 10 cities in each country, more than half of the countries (88) have fewer than 5 cities. If the MGI cities in the countries are fewer than 5, or the city to be fitted is outside the minimum or maximum of MGI population in that country, regression would be inappropriate.

Thus, in the rest 648 (15%) cases where regression is inappropriate, national GDP per capita calculated from MGI data were used together with universe population to get city GDP. For 20 cities (0.5%) whose countries do not exist in MGI, national GDP per capita data from world bank were used. 

In summary, I have used the following datasets for each part of the 4231 cities:
2617 (62%):   MGI city GDP
966   (23%):   MGI city population, population from the universe
628   (14%):   MGI city GDP, MGI city population, population from the universe
20     (0.5%):  National GDP per capita from world bank, population from the universe,

The regressions use the MGI population as the independent variable to fit the linear model (in each country), then predict on the population size from the universe of cities to get corresponding city GDP. I took this approach as otherwise, some Europe cities go very high.

Comments

Post a Comment

Popular posts from this blog

Power-law distribution (Pareto)& Zipf's Law: connection and how to fit the distribution of global city population

-
Image
- This post counts for 3 days. - In short,  We use the histogram to outline the distribution, use rank-frequency plot (log-log plot) to identify the power-law distribution, and use maximum likelihood estimation to obtain the parameters. -  I will show 1 st , how the power-law function CDF and PDF can be derived from Zipf CDF; and 2 nd , this least square estimation by excel is wrong and how to get the right one. Historical background on the power-law distribution. - Power-law distribution is as common as normal distribution in the real world. In the 19 th century, Italian economist Pareto realized that 20% of the population owns 80% the wealth, which is the famous 80/20 rule. He later named the power-law distribution as Pareto distribution, which is what we studied in the probability course. - In the 20 th century, another main contributor, Zipf made a similar discovery in linguistics about the frequency of words used. - Besides social wealth, many things follow a hig
Read more

How to Draw Heatmap with Colorful Dendrogram

-
Image
Heatmap with Dendrogram: Color Branches and Labels This data visualization example include: * Hierarchical clustering, dendrogram, and heat map based on normalized odds ratios * The dendrogram was built separately to give color to dendrogram’s branches/labels based on clusters using  dendextend * Heatmap is made by  heatmap.2  from  gplots  using the built dendrogram * The rows are sorted by means from highest to lowest, it can be done in either the dendrogram or the  heatmap.2 * Use color palettes from  colorspace Data YRBSS (Youth Risk Behavior Surveillance System) survey data from CDC. This example plots the odds ratios of 63 behavioral questions on one question regarding disordered eating. The odds ratios are median centered by column and log2 transformed Version 1: Color both the branches and labels library (curl) # read file from google drive library (gplots) # heatmap.2 library (dendextend) # make and color dendrogram library (colorspace) #
Read more

The Weighted Standard Deviation

-
Image
I have revised the figures in the Conflicts in Colombia paper to add weighted s.e. Together with confidence intervals. Turns out to be not that complicated. Weighted s.d.: Add { geom_errorbar(aes(ymin=..., ymax=...), width = 0.2 } in ggplot2 to add the confidence interval into plots: The paper discussed how conflicts in Colombia makes cities develop more sluggish and denser. I will work more on it in the later posts. The complete R code for the figure # weighted means for global 200 sample ggplot(data=m4_mean_g_g200, aes(x=year, y=wm, group=Group, color=Group, shape=Group))+ geom_point(size= 4 )+ geom_line(size= 1 )+ geom_errorbar(aes(ymin=CI_low, ymax=CI_high), width= .2 )+ ggtitle( "Urban Density Change Over Time in 200 Cities" ) + labs(x= "Year" , y= "Average City Density" , color = "" , shape= "" ) + scale_y_continuous(limits = c( 30 , 150 ))
Read more