Exploratory Data Analysis with Pandas and Python 3.x

Most of the real-world datasets contain missing values due to various reasons. In this video, we find out how we can know whether we have missing values in our dataset using Pandas library in Python.

   •  Explore the various reasons for the missing values in datasets

   •  Understand the various Pandas functions that can be used to find the missing values

   •  Learn about the different types of missing values and how Pandas does type conversion for them

Once we have learned how to find missing values in the dataset, we move on to discussing the different ways to deal with missing values.

   •  First, we discuss why simply ignoring rows with missing values might not work

   •  Understand how we can impute missing values with measures of central tendencies

   •  Demonstrate via an example about we can fill missing values based on other columns

Now, we move on to using Pandas library to deal with missing data.

   •  Explore the df.dropna function and its various attributes

   •  Explore the various ways of filling missing values via df.fillna, df.ffill, and df.bfill

   •  Implement an example in which we fill missing values based on values in other columns

We need to apply the concepts that we have learnt in this section over the real-world Titanic Dataset.

   •  Load the Titanic Dataset and explore the various columns

   •  Find out the descriptive statistics of the dataset

   •  Impute missing values in the dataset

Sometimes we might encounter values in our dataset which are abnormally high, low, or simply weird as compared to other values in the dataset. We must understand what outliers are and what causes them to occur.

   •  Understand what outliers are

   •  Understand the causes of outliers

   •  Explore via examples, the different types of outliers

Z-scores are one of the commonly used methods to identify outliers. In this video, we understand the idea behind Z-score and how they can be used to identify outliers.

   •  Discuss what are Z-scores and what do they signify

   •  Visualize Z-scores over a normal distribution for more clarity

   •  Implement Z-scores to find outliers in a dummy dataset

Z-scores can sometimes not be very efficient since they use mean and standard deviation to detect outliers. In this video, we use a modified version of Z-score which is based on median.

   •  Understand why Z-score might fail in some cases

   •  Understand the idea of Median, Standard Deviation, and Modified Z-scores

   •  Implement an example in which we find missing values using Modified Z-scores

Finally, we also learn how to use Interquartile Range (IQR) to detect outliers in a dataset and visualize them via box plots.

   •  Explore the concept of IQR and how it can be used to identify outliers

   •  Visualize IQR and outliers over a box plot

   •  Implement an example using IQR and box plots to detect outliers

Before moving on to analyzing the various types of variables in a dataset, we must understand the different variables that might occur in a dataset.

   •  Understand what are the different types of variables

   •  Explore the different types of numeric variables

   •  Explore the different types of categorical variables

Now that we have understood the different types of variables, let’s take a look at the different ways of analyzing variables using Python.

   •  Create dummy data for our analysis

   •  Implement code for plotting different types of graphs in Python

   •  Explore the different graphs and libraries available in Python

After learning about the various graphs that we can use to explore columns in Python, we must first understand the concept of Skewness and Kurtosis in Statistics and how they affect the shape of a distribution.

   •  Understand what Skewness is

   •  Understand the idea behind Kurtosis

   •  Explore how Skewness and Kurtosis affect the shape of the curve

Finally, we will apply the different techniques that we have learned for Univariate Analysis over the Olympics Dataset.

   •  Explore the different columns in Olympics Dataset

   •  Draw density plots, histograms, and so on. over various columns

   •  Find Skewness of the data using SciPy module in Python

Now that we have explored univariate analysis, we move ahead to bivariate analysis where we explore two variables at the same time.

   •  Understand what is bivariate analysis

   •  Understand how bivariate analysis helps us understand our data better

   •  List out various graphs used for bivariate analysis

Before moving on to doing practical bivariate analysis, we must understand the theoretical concept behind correlation coefficients.

   •  Explore the concept of correlation coefficient

   •  Understand the different types of correlation coefficient

   •  Understand what correlation coefficient signifies for our data

After understanding the theoretical concepts behind correlation coefficients, we now move on to visualizing correlation between two sets of variables.

   •  Implement code for positive and negative correlation

   •  Use seaborn library to visualize scatterplots

   •  Use heatmaps to visualize correlation between multiple pair of columns at once

In this video, we will apply various techniques of bivariate analysis over the Titanic Dataset.

   •  Load the Titanic Dataset

   •  Implement bivariate graphs using Seaborn

   •  Identify trends if they exist in the data

In this video, we will apply various techniques of bivariate analysis over the video game sales dataset.

   •  Load the video game sales dataset and understand the various columns

   •  Implement interactive graphs using Bokeh library in Python

   •  Identify trends if they exist in the data using bivariate graphs

Now that we have explored univariate and bivariate analysis, we move ahead to multivariate analysis where we explore more than two variables at the same time.

   •  Understand what is multivariate analysis

   •  Understand the various advantages of multivariate analysis

   •  Visualize a graph depicting multivariate analysis

In this video, we will apply various techniques of multivariate analysis over the Titanic Dataset.

   •  Load the Titanic Dataset and find descriptive statistics of the various variables

   •  Implement multivariate graphs using Seaborn

   •  Identify trends if they exist in the data

In this video, we will apply various techniques of multivariate analysis over the Pokemon Dataset.

   •  Load the Pokemon Dataset and find descriptive statistics of the various variables

   •  Implement interactive graphs using Bokeh

   •  Identify trends if they exist in the data using multivariate graphs

Simpson’s Paradox is a phenomenon that may occur in real-world data, leading to conflicting results. We understand why it happens and what we can do to prevent it.

 •   Understand what is Simpson’s Paradox

 •   Understand what causes it and how we can prevent it from happening

   •  Demonstrate Simpson’s Paradox using an example

This is one of the most widely misinterpreted phenomena that occurs in real world. We understand why it happens and what we can do to prevent it.

 •  Understand why Correlation does not necessarily imply causation

 •  Understand what causes it and how we can prevent it from happening

 •  Demonstrate that correlation does not imply causation using various examples

In this video, we will apply all the different techniques that we have learned in the previous sections to a real-world dataset.

 •  Download and load the dataset

 •  Explore the different variables in the dataset

 •  Create a set of questions that we will answer through our analysis

Here we will do Exploratory Data Analysis over Red Wine Data.

 •  Download and load the dataset

 •  Explore the different variables in the dataset

 •  Identify trends if they exist in the data

In this video, we will do Exploratory Data Analysis over White Wine Data.

 •  Download and load the dataset

 •  Explore the different variables in the dataset

 •  Identify trends if they exist in the data

Here, we will do a comparative analysis about how these wines are different from each other.

 •  Download and load the dataset

 •  Explore the different variables in the dataset based on the type of wines

 •  Identify trends if they exist in the data