Global AI and Data Science

The goal of EDA is to leverage visualization tools, summary tables, and hypothesis testing to:

• Provide summary level insight into a dataset.
• Uncover underlying patterns and structures in you data.
• Identify outliers, missing data, class balance, and other data-related issues.
• Relate the available data to the business opportunity.

Let’s work with a case study that comes from the online retail data set and are available through the UCI Machine Learning Repository. This is a transnational data set which contains all the transactions occurring between 01/12/2010 and 09/12/2011 for a UK-based and registered non-store online retail.The company mainly sells unique all-occasion gifts. Many customers of the company are wholesalers.

Business scenario here is that the management team expects to spend less time in projection models and gain more accuracy in forecasting revenue. It is expected that well projected numbers will help stabilize staffing and budget projections which will have a beneficial ripple effect throughout the company.

Business metric can be defined as a function of revenue gained through more accurate predictions.

Steps of the EDA Process:

1. Load data into pandas, NumPy or another similar tool and summarize the data 

   

Loading data into pandas

2. Use tables, text and visualizations to tell the story that relates the business opportunity to the data

Monthly Revenue Calculation

Here, the data is leveraged to calculate the monthly revenue of the online retail store. Since one of the goals of this case study is forecasting revenue, therefore it is important to quantify revenue using such formulae which can later be utilized either in supervised learning or for hypothesis testing.

3. Identify a strategy to deal with missing values:

It is during the Exploratory Data Analysis (EDA) process that data integrity issues are identified sometimes. After extracting data it is important to include checks for quality assurance even on the first pass through the project workflow. Quality assurance step must implement checks for duplicity and missing values. Missing values are generally dealt with depending on the category of missingness i.e MCAR (Missing completely at random), MAR (Missing at random) and MNAR (Missing not at random). If the missing data are not MCAR, then imputing values can result in an increase in bias and therefore it is very important to have train/test split.

Data cleaning summary

4. Investigate the data and underlying business scenario with visualizations and hypothesis testing.

Jupyter notebook is predominantly used for investigating the data with visualizations. However, best practices as a data scientist generally require that maximum amount of code is saved as text files either in simple scripts, modules, or Python package. This ensures re-usability, allows for unit testing and works naturally with version control.

For example, plot_rev() function has been called from the python script data_visualization.py rather than writing scripts for plotting repetitively in the Jupyter notebook.

EDA charts with plot_rev() function

Jupyter notebook should be kept as a presentable component with minimal code. It can used as a Data Scientist’s powerpoint to deliver your story of the initial findings on the data.

5. Communicate your findings

There is no single right way to communicate EDA, but a minimum bar is that the data summaries, key findings, investigative process, conclusions are made clear. Deliverables should be concise and clear.

One important deliverable could be the result of Investigating the relationship between the relevant data, the target and the business metric. For example, revenue calculated in step 2 could be the target variable directly related to the business metric and a proposal for supervised learning and/or forecasting model could be substantiated with the EDA deliverable.

Follow this link to access the notebook for EDA