Week 3/19 How many jobs are passed on the way in Chicago?

-
Revised on Mar.25th, calculation of jobs passed was not correct.

3.21 - 23 (Fri)
Since Wednesday I have been working on a problem that looks similar to the global conflict score I calculated before. We have the coordinates of both start points (home) and end points (work) of 3.5 million commuting trips in Chicago. One trip can carry more than one person, the total number of people on all the trips is 3.7 million. So each trip is weighted, but the average number of persons on each trip is merely 1.07.

1. Most trips in the dataset were done by only one person

The distribution of the weights: number of people on each trip between each pair of origin and destination are highly skewed to the right: the busiest trip had 109 persons traveling from Indian Village to the University of Chicago. Actually, the top 13 trips (853 persons) all target somewhere in the University of Chicago (coordinate: 41.78937, -87.60285). On the other hand, 95% of the 3.5 million trips have only 1 person, 99.6% of the trips have 3 or fewer persons.

2. The average travel distance for all the trips is 19 km.

It is easy to get the travel distance for all the 3.5 million trips. The mean is 19.728 km, the weighted mean is 19.181 km, the median is 15.262 km, sd = 15.98 km, min = 0 km, and max = 141 km.

(Notice that statistics on distance-related variables obtained from a 1% random sample is quite representative of the population, with mean = 19.7, weighted man = 19.18)

Visualization of  35,040 (1%) randomly sampled trips: (longer trips are marked in yellow, shorter trips are marked in red.)


  • See next post for technical details about how this is done using ggmap in R.
  • See this post for further discussions on how to use heatmap to produce a proper visualization on home and job densities.

* The histogram of travel distance, skewed to the right not too extreme.


 * People living in the peripheral do travel longer distances
Visualization of travel distance for people from home on the map of Chicago. Darker red shows places where people living there travel longer distances:

Besides a heat map, I also tried 3d surface using wireframe function from R package lattice. I attached the code in the next post.


Not too surprising, most people work close to home. But those living on the periphery will travel a longer distance. People living in city center travel relatively shorter trips.
* People working in the city center do not necessarily travel a long distance
On the other hand: Do people working in city center travel longer? It doesn't seem to be so.
Visualization of travel distance for people at work on the map of Chicago. Dark red shows places where people work there travel long distances:

3. How many jobs have been passed to reach one's job?

To measure the integration of the job market in the city, we are interested in the number of opportunities bypassed by each passenger to reach his job (a foraging index). In another word, we suppose that if someone travels a long distance, it is because he doesn't want or cannot get the nearby jobs. 
The distribution is moderately right-skewed.
The maximum number of jobs passed is close to total jobs in the sample. Mean = 7,649 (jobs), weighted mean = 8,418 (jobs), median = 4,944, min = 0, max = 37,240, sd = 9,298.  
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
      0    1091      4944       8685   13771     37240 

Based on 350 K (10%) random sample which contains 373 K jobs:
Every statistic is roughly 10 times the results from 1% sample. 
   Min. 1st Qu.   Median    Mean   3rd Qu.     Max. 
      0   10765     49491      86594  136634     373094 

So the percentage of job passed by a traveler on average are the same: 86,594 / 373,000 = 23%.

4. Challenges during programming 

I adopt the similar code to calculate the number of jobs passed as the one for conflict score. I wrote some distance function and used mapply in r to calculate for each person (each point), its distances to all the other points, and sum those within his distance to work. But this time there are 3.5 million locations to go through. (conflict score was on a scale of < 10k). I anticipated that the code I wrote might take too long since running on a testing subset of 10k takes several minutes. I first tried to improve the program, which did not improve anything. Then I tried packages written by others to get distance matrix then timed the vector of weights to obtain the vector of "job passed" directly, which looks neat but takes much longer when the matrix size is over 10k.

(Comments 3.27) Now calculating 350 K records took 7 hours. 3.5 M would take maybe a month. So the limit of this approach is about 1 million trips. 

--------------------------------------
3.19 (Mon) Re-ran all the figures for the Colombia paper
3.20 (Tue) Used R markdown to generate reports for road and block

Comments

Post a Comment

Popular posts from this blog

How to Draw Heatmap with Colorful Dendrogram

-
Image
Heatmap with Dendrogram: Color Branches and Labels This data visualization example include: * Hierarchical clustering, dendrogram, and heat map based on normalized odds ratios * The dendrogram was built separately to give color to dendrogram’s branches/labels based on clusters using  dendextend * Heatmap is made by  heatmap.2  from  gplots  using the built dendrogram * The rows are sorted by means from highest to lowest, it can be done in either the dendrogram or the  heatmap.2 * Use color palettes from  colorspace Data YRBSS (Youth Risk Behavior Surveillance System) survey data from CDC. This example plots the odds ratios of 63 behavioral questions on one question regarding disordered eating. The odds ratios are median centered by column and log2 transformed Version 1: Color both the branches and labels library (curl) # read file from google drive library (gplots) # heatmap.2 library (dendextend) # make and color dendrogram library (colorspace) #
Read more

Power-law distribution (Pareto)& Zipf's Law: connection and how to fit the distribution of global city population

-
Image
- This post counts for 3 days. - In short,  We use the histogram to outline the distribution, use rank-frequency plot (log-log plot) to identify the power-law distribution, and use maximum likelihood estimation to obtain the parameters. -  I will show 1 st , how the power-law function CDF and PDF can be derived from Zipf CDF; and 2 nd , this least square estimation by excel is wrong and how to get the right one. Historical background on the power-law distribution. - Power-law distribution is as common as normal distribution in the real world. In the 19 th century, Italian economist Pareto realized that 20% of the population owns 80% the wealth, which is the famous 80/20 rule. He later named the power-law distribution as Pareto distribution, which is what we studied in the probability course. - In the 20 th century, another main contributor, Zipf made a similar discovery in linguistics about the frequency of words used. - Besides social wealth, many things follow a hig
Read more

eXtreme Gradient Boosting (XGBoost): Better than random forest or gradient boosting

-
Image
Ensemble Methods 1. Random Forest 2. Gradient boosting 3. Extreme Gradient Boosting Compared to Lasso and Best-subset Lasso Best-subset Compare MSEs Code I recently had the great pleasure to meet with Professor Allan Just and he introduced me to eXtreme Gradient Boosting (XGBoost). XGBoost is “one of the most loved machine learning algorithms at Kaggle”, it somehow combines the advantages of random forest and boosting. We will see later its predictive ability is indeed very strong. In the earlier post  Model Selection using Lasso and Best Subset  I mainly focused on model selection by best-subset, stepwise, or lasso. This post will focus on ensemble method: random forest, Gradient boosting, and eXtreme Gradient Boosting (XGBoost). An ensemble method is a technique that combines the predictions from multiple machine learning algorithms together to make more accurate predictions than any individual model. 1. Random Forest Use library  randomForest . A
Read more