Automatidata project #

Course 2 - Get Started with Python

Course 2 end-of-course project: Inspect and analyze data #

In this activity, you will examine data provided and prepare it for analysis.

The purpose of this project is to investigate and understand the data provided.

The goal is to use a dataframe contructed within Python, perform a cursory inspection of the provided dataset and inform team members of your findings.

This activity has three parts:

Part 1: Understand the situation

• How can you best prepare to understand and organize the provided taxi cab information?

Part 2: Understand the data

• Create a pandas dataframe for data learning, and future exploratory data analysis (EDA) and statistical activities.

• View and interpret the datasets data table

• Sort and interpret the data table for two variables of your choice.

Part 3: Understand the variables

• What is the min, mean, and max of your chosen variables?

• Visualize the variables

Be sure to complete this activity before moving on. The next course item will provide you with a completed exemplar to compare to your own work.

Identify data types and relevant variables using Python #

Exercise Instructions: #

Complete the following step-by-step instructions to inspect and analyze this NYC taxi dataset.

This activity will help ensure the information is:

1. Ready to answer questions and yield insights

2. Ready for visualizations

3. Ready for future hypothesis testing and statistical methods

Follow the instructions and answer questions to complete this activity. Afterward,

1. Write a short Executive Summary using your findings.

2. Use the structured notebook provided to help you in this project. Please complete the questions inside and prepare a summary for the data team.

3. Consider the questions presented in the Course 2 PACE strategy document.

4. Compare your data insights with the provided exemplar to confirm of your approach and results.

Throughout these project notebooks, you’ll see references to the problem-solving framework PACE. The following notebook components are labeled with the respective PACE stage: Plan, Analyze, Construct, and Execute.

PACE: Plan #

Step 1a. Understand the situation #

1. How can you best prepare to understand and organize the provided taxi cab information?

Begin by exploring your dataset and consider reviewing the Data Dictionary.

### **==> EXEMPLAR CODE and OUTPUT**
#
# Question 1: How best to prepare?
#
# Q1 Answer:
# By reading the taxi cab data fields and ensuring I understand how 
# each one impacts the dataset. Reviewing the fact sheet could also 
# provide helpful background information. 

# However, my primary goal is to get the data into Python, inspect it, 
# and provide DeShawn with my initial observations. 
# Afterwards, I can learn more deeply about the data and check
# for any anomalies.

Step 1b. Import dataframe #

Create a pandas dataframe for data learning, and future exploratory data analysis (EDA) and statistical activities.

Code the following,

• import pandas as pd #library exercise for buidling dataframes

• import numpy as np #numpy is imported with pandas

• import matplotlib.pyplot as plt #visualization library

• import seaborn as sns #visualization library

• df = pd.read_csv(‘2017_Yellow_Taxi_Trip_Data.csv’)

Note: pair the data object name “df” with pandas functions to manipulate data, such as df.groupby().

# ==> EXEMPLAR CODE and OUTPUT
import pandas as pd               #library exercise for buidling dataframes
import numpy as np                #numpy is imported with pandas
import matplotlib.pyplot as plt   #visualization library
import seaborn as sns             #visualization library

df = pd.read_csv('2017_Yellow_Taxi_Trip_Data.csv')
print("done")

done

PACE: Analyze #

Step 2a. Understand the data - Interpret the data table #

View and interpret the datasets data table by coding the following:

1. df.head(10)
2. df.info()

Consider the following two questions:

Question 1: When reviewing the df.head(10) output, are there any data points that surprise you or are not correct?

Question 2: When reviewing the df.info() output, what kind of data types are we working with?

# ==> EXEMPLAR CODE and OUTPUT
df.head(10)

# ==> EXEMPLAR CODE and OUTPUT
df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 22699 entries, 0 to 22698
Data columns (total 18 columns):
 #   Column                 Non-Null Count  Dtype  
---  ------                 --------------  -----  
 0   Unnamed: 0             22699 non-null  int64  
 1   VendorID               22699 non-null  int64  
 2   tpep_pickup_datetime   22699 non-null  object 
 3   tpep_dropoff_datetime  22699 non-null  object 
 4   passenger_count        22699 non-null  int64  
 5   trip_distance          22699 non-null  float64
 6   RatecodeID             22699 non-null  int64  
 7   store_and_fwd_flag     22699 non-null  object 
 8   PULocationID           22699 non-null  int64  
 9   DOLocationID           22699 non-null  int64  
 10  payment_type           22699 non-null  int64  
 11  fare_amount            22699 non-null  float64
 12  extra                  22699 non-null  float64
 13  mta_tax                22699 non-null  float64
 14  tip_amount             22699 non-null  float64
 15  tolls_amount           22699 non-null  float64
 16  improvement_surcharge  22699 non-null  float64
 17  total_amount           22699 non-null  float64
dtypes: float64(8), int64(7), object(3)
memory usage: 3.1+ MB

Step 2b. Understand the data - Sort by variables #

Sort and interpret the data table for two variables of your choice.

Answer the following three questions:

Question 1: Sort your first variable (trip_distance) from maximum to minimum value, do the values seem normal?

Question 2: Sort your by your second variable (total_amount), are any values unusual?

Question 3: Are the resulting rows similar for both sorts? Why or why not?

# ==> EXEMPLAR CODE and OUTPUT

df_sort = df.sort_values(by=['trip_distance'],ascending=False) 
df_sort.head(10)

# ==> EXEMPLAR CODE and OUTPUT
df_sort = df.sort_values(by=['total_amount'],ascending=False)
df_sort.head(10)

Exemplar response: The two outputs in the table show different rows of data. A reason for this observation might be because there are trips that have a shorter distance, yet higher total amount (total ride fare).

Step 2c. Understand the data - Use df.describe() #

Instructions: Using pandas df.describe(), what is the min, mean, and max of your chosen variables?

Code and perform the following:

• df.describe()

• Question 1: What is the min, mean, and max of your first variable?

• Question 2: What is the min, mean, and max of second variable?

• Question 3: Are the values easily readable? Would could be done to make them more easily readable?

# ==> EXEMPLAR CODE and OUTPUT

df.describe()

#Question 1: what is the min, mean and max of your first variable ?

#Answer:             Min        Mean    Max
#total_amount        -120.30    16.31   1200.29

#Question 2: what is the min, mean, and max of your second variable?

#Answer:               Min   Mean    Max
#trip_distance         0     2.91    33.96

PACE: Construct #

Step 3a. Visualize you variables #

Instructions: Create a histogram for each of the two variables. Act and reflect on the following steps:

1. Histogram of your first variable (total_amount)

2. Histogram of your second variable (trip_distance)

3. Are your variables numerical (did the code work)?

# ==> EXEMPLAR CODE and OUTPUT
plt.figure(figsize=(8,6))
plt.xticks(fontsize=14); plt.yticks(fontsize=14)
df = df.sort_values(by='total_amount')
plt.hist(df['total_amount'],bins=[0,20,40,60,80,100])
plt.title('Histogram of Total Amount',fontsize=20)
plt.xticks(fontsize=15); plt.yticks(fontsize=15)
plt.xlabel('$ amount bin',fontsize=20)
plt.ylabel('Count', fontsize=20)

Text(0, 0.5, 'Count')

# ==> EXEMPLAR CODE and OUTPUT
plt.figure(figsize=(8,6))
#plt.xticks(fontsize=14); plt.yticks(fontsize=14)
df = df.sort_values(by='trip_distance')
plt.hist(df['trip_distance'],bins=[0,5,10,20,25])
plt.title('Histogram of Trip Distance',fontsize=20)
plt.xticks(fontsize=15); plt.yticks(fontsize=15)
plt.xlabel('Trip distance bin',fontsize=20)
plt.ylabel('Count', fontsize=20)

Text(0, 0.5, 'Count')

PACE: Execute #

Given your efforts, what can you summarize for DeShawn? #

Note for Learners: Your answer should address Luana’s request for a summary that covers the following points:

• A summary of the data type of each variable

• Any relevant and irrelevant columns

• The minimum, mean, and max for the two most relevant variables

• A summary of the data visualization