Visualization of commuting connections in Chicago

-
Following the last post, naturally, we would like to observe these 3.5 million connections directly. But even with 35k connections, the lines can fill the whole map. It makes more sense to show randomly sampled connections while we have checked that the sample is pretty representative of the parent distribution.
This post includes the R code in the end.

Connection Map

The main observation is that shorter trips are more clustered near the city center.

Fig 1. 35 K trips (1% of the 3.5 M), shorter trips marked in red and longer trips marked in yellow.
(If mark longer trips with the darker color they would cover everything beneath. )

Fig 2. Use only 3.5 k trips (1/10 of fig 1), red color shows shorter trips.


Fig 3. Use only 3.5 k trips, but shorter trips are less transparent (higher alpha value)
The less transparent (longer) trips can hardly be identified since there are too many short trips covering on top.


Density Map

The relative density of where do people live:


And where do people work:

Job opportunities are relatively more concentrated in the center than locations of homes.

Method

Some nice examples online use bezier curves to show air connections in the United States or in the world:
Example by Nathan Yau
More ggmap examples
My earlier post on global air connections

But to show connections within a city, using bezier curve is not necessary because directly line can serve the purpose well enough in such a small area. And the for loop written by Nathan Yau might encounter a problem when the number of lines increases. To plot thousands of lines onto a map. First use reshape to transpose the original [home_long, home_lat, work_long, work_lat] data frame from wide into long format: [long, lat], but grouped by each home - destination pairs. So for N = 35k, there are 35k groups: [1,1, 2,2, 3,3, .... , 35k, 35k]. Then use ggmap and ggplot to add lines by the groups.
# first, sort data set by travel distances (DisTravel) from long to short, then assign groups to each observation (a home - work cooridnate pair). So later longer trips will be assigned a deeper red color.
test <- test[order(test$DisTravel, decreasing = T),]
n_row <- nrow(test)
test$group <- 1:n_row

# create a color scale of 100 colors to all the groups:
# long distance (yellow) -> short distance (red)
pal <- colorRampPalette(c("yellow", "red"))
colors <- pal(100)
colors_panel <- colors[ceiling(test$group/n_row*length(colors))]

# use reshape to convert from wide to long
test_long <- reshape(test, direction = 'long',
                     varying = list(
                       c('h_cen_long', 'w_cen_long'),
                       c('h_cen_lat', 'w_cen_lat')
                     ),
                     timevar = 'location',
                     times = c('home','work'),
                     v.names = c('long', 'lat')
)

library(ggmap)
library(ggplot2)
chicago_map <- get_map(location = 'chicago_map', zoom = 9)

# add lines with manual color, remove legend:
ggmap(chicago_map)+
  geom_line(data = test_long,  alpha = 0.2, lwd = 0.5,
            aes(x=long, y=lat, group=group, color=factor(group))) +
  scale_color_manual(values=colors_panel) +
  theme(legend.position="none",
        axis.title.y = element_blank(), axis.title.x = element_blank()) 
###



As a comparison, using bezier curves by adopting code from Nathan Yau shows a similar result:


I discussed heat map (and code) in the next post.




The code for 3d surface map:
# For 3d surface, the dataset has 3 variables: Measurement, long, lat. 
library("lattice")
# A single figure example for testing: 
wireframe(DisTravel ~ long * lat, data = homeBlock2,
          main = "Average Distance Travelled by Home Block",
          zlab = "Distance (km)",
          drape = TRUE, # add color 
          colorkey = F, # add color key
          pretty=T,     # not so useful in this case
          # adjust how to view the screen 
          screen = list(z = -80, x = -60), alpha = 0.6
)

# Draw two figures in one frame:
par.set <- list(axis.line = list(col = "transparent"),
           clip = list(panel = "off"))
# Ave Distance travelled by home block
print(wireframe(DisTravel ~ long * lat, data = homeBlock2, 
                main = "Distance Travelled by Where People Live",
                zlab = "Distance (km)",
                drape = TRUE, colorkey = TRUE,   
                screen = list(z = 0, x = -60),
                par.settings = par.set
                ),
      split = c(1,1,2,1), more = TRUE)
print(wireframe(DisTravel ~ long * lat, data = homeBlock2, 
                zlab = "Distance (km)",
                drape = TRUE, colorkey = TRUE, 
                screen = list(z = -80, x = -60),
                par.settings = par.set
                ),
      split = c(2,1,2,1))


Comments

Post a Comment

Popular posts from this blog

How to Draw Heatmap with Colorful Dendrogram

-
Image
Heatmap with Dendrogram: Color Branches and Labels This data visualization example include: * Hierarchical clustering, dendrogram, and heat map based on normalized odds ratios * The dendrogram was built separately to give color to dendrogram’s branches/labels based on clusters using  dendextend * Heatmap is made by  heatmap.2  from  gplots  using the built dendrogram * The rows are sorted by means from highest to lowest, it can be done in either the dendrogram or the  heatmap.2 * Use color palettes from  colorspace Data YRBSS (Youth Risk Behavior Surveillance System) survey data from CDC. This example plots the odds ratios of 63 behavioral questions on one question regarding disordered eating. The odds ratios are median centered by column and log2 transformed Version 1: Color both the branches and labels library (curl) # read file from google drive library (gplots) # heatmap.2 library (dendextend) # make and color dendrogram library (colorspace) #
Read more

Power-law distribution (Pareto)& Zipf's Law: connection and how to fit the distribution of global city population

-
Image
- This post counts for 3 days. - In short,  We use the histogram to outline the distribution, use rank-frequency plot (log-log plot) to identify the power-law distribution, and use maximum likelihood estimation to obtain the parameters. -  I will show 1 st , how the power-law function CDF and PDF can be derived from Zipf CDF; and 2 nd , this least square estimation by excel is wrong and how to get the right one. Historical background on the power-law distribution. - Power-law distribution is as common as normal distribution in the real world. In the 19 th century, Italian economist Pareto realized that 20% of the population owns 80% the wealth, which is the famous 80/20 rule. He later named the power-law distribution as Pareto distribution, which is what we studied in the probability course. - In the 20 th century, another main contributor, Zipf made a similar discovery in linguistics about the frequency of words used. - Besides social wealth, many things follow a hig
Read more

eXtreme Gradient Boosting (XGBoost): Better than random forest or gradient boosting

-
Image
Ensemble Methods 1. Random Forest 2. Gradient boosting 3. Extreme Gradient Boosting Compared to Lasso and Best-subset Lasso Best-subset Compare MSEs Code I recently had the great pleasure to meet with Professor Allan Just and he introduced me to eXtreme Gradient Boosting (XGBoost). XGBoost is “one of the most loved machine learning algorithms at Kaggle”, it somehow combines the advantages of random forest and boosting. We will see later its predictive ability is indeed very strong. In the earlier post  Model Selection using Lasso and Best Subset  I mainly focused on model selection by best-subset, stepwise, or lasso. This post will focus on ensemble method: random forest, Gradient boosting, and eXtreme Gradient Boosting (XGBoost). An ensemble method is a technique that combines the predictions from multiple machine learning algorithms together to make more accurate predictions than any individual model. 1. Random Forest Use library  randomForest . A
Read more