Part 6.2: Predictive Model - Including Date

• Next steps will be to factor in speed and distance based on Gender and Trip. By not being able to encode start and end stations (due to the sheer number of points), we are losing crucial information on the trip itself. We need another proxy for those measures.
• Another change I could make is to bin age into buckets. However, the data indicates that age has no correlation or effect on the trip duration. This is counter-intuitive, however, I don't have a good reason to refute the data.

1. Include Average Speed based on: Trip (Start + End Station) and User Type. 
   1. Reason for Trip: Some trips are up hill, others are down hill. Some routes, such as one through times square involve heavy traffic, based on intuition.
   2. Reason for User Type: Tourists (Customers), will usually ride more slowly with frequent stops than a Subscriber, according to the data.

2. Include average duration for each trip based on: Trip and User Type for reasons mentioned above

def get_speed_distance(df):

    df['Start Station Name'] = df['Start Station Name'].astype(str)
    df['End Station Name'] = df['End Station Name'].astype(str)
    df['Trip'] = df['Start Station Name'] + ' to ' + df['End Station Name']
    df['Trip'] = df['Trip'].astype('category')

    df['avg_speed'] = df.groupby(['Trip','User Type'])['mile_hour'].transform('mean')
    df['avg_duration'] = df.groupby(['Trip','User Type'])['Trip Duration'].transform('median')

    return df

• The model is significantly better. But can it be even better?
• One factor which would have a significant impact on trip duration is traffic. However, since we do not know what route the rider took it's difficult to incorporate this information. Lastly, the Google Maps API caps usage, so we can't use it to identify traffic patterns easily.

• One factor which a lot of people think may be a good predictor of trip duration is weather. I personally disagree. Weather influences wether or not a user will bike, not how long they will bike. If it's snowing, I won't bike to work. If it's nice, I will bike to work. Regardless of my opinion, I am going to test this hypothesis. If weather is not a strong indicator, I will remove it in the next model.
• The weather data was acquired from the National Centers for Environmental Information. The data sourced from the website is a daily summary. The attributes include: high temperature (F), low temperature (F), and precipitation (inches).

def get_weather(df):
    df['DATE'] = to_datetime(df['Start Time'].dt.date)
    df = df.merge(df_weather, on = 'DATE', how = 'left')
    return df

• Little to no improvement in the model. Weather has little to no impact. 
• The coefficient for avg_duration spiked up. Not sure why, but the higher coefficient makes sense since duration is our target variable, and avg_duration is a solid proxy (anchor) for how long a trip most likely takes. 
• Let's confirm the effectiveness of the model with cross validation:

CV accuracy: 0.874 +/- 0.001

• Based on some of the observations above I ran the same model above without weather and date information as predictors.  
• As we can see, there's a very small decrease in R². Another interesting observation is the effect of this data on the coefficient of user type, it's almost halved.
• The effect on the random sample is small, however, with 10 times the amount of data, the effect could be slightly larger, so for the final model, let's keep the date information.
• Lastly, I chose to test other regression algorithms such as random forests to see if another regressor performed better. I chose to keep the n_estimators low due to run-time concerns. With this parameter set to 80, the model took 10 minutes to run with an R² the same as that of the linear regression. It could be worth pursuing random forests by optimizing other parameters such a min_samples_leaf. One approach may be to see the distribution of trip duration to identify the number of of trips which take 5–6 minutes, 6–7 minutes, etc. This could help identify the min_samples_leaf parameter. In the real world, we don't need to be accurate to the exact second. As long as we can predict how long the trip will take within a minute, it's a solid result in my opinion. Google maps doesn't tell you how long your trip will take to the exact second, it gives you it's prediction as an integer in minutes. 

Part 6.5: Predictive Model - Final Model

• I could've used XGboost and other fancy algorithms, however, for a dataset of this size, it would take too long to run and the gains wouldn't be worth it if there are any.
• Final Model: Linear Regression (worth exploring Lasso)
  • Predictors: Distance, Gender, Average Duration and Average Speed based on Trip and Gender, User Type, and Date information

CV accuracy: 0.852 +/- 0.000

*The code is a work in progress and constantly undergoing changes and improvement with your feedback. Please leave your thoughts in the comments section below. Thank you.