Statistics and Data Analysis

Projecting data onto geological grid using spatial join in R We have lots of observations and their coordinate (lat, long), and I want to group these observations by a manually designed grid. In the beginning, I wrote a for loop which is super slow (about  5 hours for 3.5 million observations). Then I was reminded that there is something called spatial join. In Python, it is done by  gpd.sjoin() .  as Xinyue told me. In R we can use over() from sp , which is a package that provides classes and methods for spatial data: points, lines, polygons, and grids. Spatial join toke fewer than  2 minutes . The  over()   function is not very easy to use, I found out one way to get it work by turning grid into Spatial Polygon object first. There are different approaches to do this.  The main dataset of our observations has recorded millions of trips: the longitudes and latitudes of the trip origin, and the counts of people on those trips by social eco...

As well as how the distances between geological points were calculated Calculating the distance between two points on the earth is fundamental to calculate all the measurements mentioned in earlier posts. The distance in kilometers is calculated using the  Spherical Law of Cosines : d = acos( sin(lat1)*sin(lat2) + cos(lat1)* cos (lat2)* cos (long2-long1) )  * 6371 (km) Step-by-step explanation: The sin and cos functions take latitudes and longitudes input in radians in most softwares.  For example, Shanghai has a coordination of ( 31.222, 121.458 ), its (lat1, long1) in radians is ( 31.222/180 * pi, 121.458/180 * pi) = (0.55, 2.12), pi = 3.14159... The coordination for New York is (40.717, -74.004), in radian: (0.71, -1.29). sin(lat1)* sin (lat2) +  cos (lat1)* cos (lat2)* cos (long2-long1)  =  sin ( 0.55 )* sin ( 0.71 ) +  cos ( 0.55 )* cos ( 0.71 )* cos (-1.29 - 2.12) = -0.28  ---> this output is a value from -1 to 1 Th...