Data Wrangling with Python 3.x

Sometimes organizations/companies find it convenient to store the relevant information about something in sheets of Excel file. We will look into xlrd module to extract information out of Excel file.

• Analyze the dataset by manually opening the file

• Import the dataset into Python using xlrd module and then print the sheet names and number of sheets

• Print the rows of a sheet on the console

People prefer a small portion of code to get big things done so we will use Pandas module in this video to do the same.

• Import the Pandas module

• Import the different sheets of Excel files into the Pandas DataFrames and extract some basic information

Sometimes a need arises to extract information out of the PDF files and then process that. We are going to look how we can do that in this video.

   •  Extract information out of a PDF file and then store each page of PDF file in  a separate index of a list

   •  Print each page of PDF files

Sometimes we feel a need to write the data to a PDF file, so in this video we will look how to edit to a PDF file.

   •  Construct a sample resume in the code example

   •  Edit text and images to a PDF file at proper positions

Database administrators choose their databases based on the characteristics of databases. We will just look into the basics.

   •  Understand when to choose relational database and when to choose the non-relational database

   •  Look at the links containing the software’s required to install in this section

We will be storing the JSON file into SQLite light weight database and look into the code example to accomplish that.

   •  Create the table containing fields from the JSON file in SQLite

   •  Dump  the JSON file by parsing it into the SQLite databases

   •  Verify the dump using DB Browser for SQLite

Many of the times in industry people prefer non-relational databases over relational databases due to complexities of schemas. We will dump information in MongoDB (a famous document oriented database).

   •  Make MongoDB up and running

   •  Write the code to dump the CSV file into MongoDB

   •  Verify the dump using Robo3T(Robomongo)

In this video we are going to use Elasticsearch with Kibana(to display information from Elasticsearch) to store the JSON file into the Elasticsearch.

   •  Explore Elasticsearch and Kibana

   •  Import the CSV file and then convert each to a format which can be dumped to Elasticsearch

   •  Verify the dump using Kibana

Often people are interested in the pros and cons of the databases, so in this video we are going to look into that in detail.

   •  Understand advantages and disadvantages of relational and non-relational databases

Data cleansing holds an important part in Data Science. We will look into why it is important and some common tips and methods to do it.

   •  Explore importance of data cleansing in Data Science

   •  Learn about data cleansing tips and techniques

We are going to jump into looking data frames and what they are, and how they display structured information in a good format.

• Read datasets, displaying column names, displaying the number of rows and number of columns in a data frame

• Changing the data type of columns and retrieving certain rows from the data frames

Sometimes there is a need to give proper names to columns coming in the datasets, adding more and removing the irrelevant ones so this video will show you how to perform that.

   •  Edit the column names

   •  Delete the irrelevant columns

   •  Add more columns into the data frame

Duplicate rows based on a column values might be redundant in performing the operations in Data Science so it is good to drop them.

   •  Drop all the duplicate rows

   •  Drop all but keep the first duplicate row

   •  Drop all but keep the last duplicate row

Sometimes we need to extract just required columns and rows out of the data frames. In this video we are going to look at the lines of codes which can be used to do so.

   •  Read in the dataset and then retrieving the first, last rows and columns

   •  Retrieve the first five columns and first five rows

   •  Retrieve certain rows and columns

Data distributed in different sheets can be concatenated and merge/join depending on the use case. In this video we are going to solve this issue which occurs a lot in industry.

   •  Look at the syntax of how we can create a data frame from a dictionary

   •  Concatenate data frame in Python

   •  Perform join/merge operation between two data frames on a column

Real time datasets contains many missing values in columns. We will look into this video how we can solve this problem and come to a good solution.

• Understand how the missing values appears in the dataset and also learn how to deal with them

• Introduce some missing values in a data frame

• Drop the rows/columns which contain missing values or variables or mean of the column in which they are present

Analyzing the dataset sometimes require rows to be sorted. Also we need to filter out some rows based on various conditions. We will look into this video how we can do that.

   •  Edit the certain columns for the sake of computation

   •  Sort the data frame on a column and look into the syntax of how we can do that

   •  Filter out the certain conditions and look into the syntax of how we can do that

Computers understand numbers. Sometimes machine learning algorithms require columns to be in equivalent numeric form so we will look in this video how we can do that.

   •  Drop the rows which are containing the missing values

   •  Use LabelEncoder from the pre-processing module to encode the gender column values

   •  Add the encoded gender column back into the data frame

We will look into another technique of mapping more than two unique values in a column.

   •  Drop the rows having the missing values and get the unique values in a column

   •  Construct a dictionary mapping those unique values to different values

   •  Apply the new encoding onto the column of data frame and look at the changes

Rescaling is mapping the numeric values in a column to (0 to 1) range and it helps machine learning algorithms to converge faster. Standardization helps to map column values in such a way that they have mean of zero and standard deviation of one. This helps to compare feature along different scales.

• Rescale a column using the MinMaxScaler of pre-processing module and this we will look into the results

• Standardize a column using the StandardScaler class of pre-processing module. And then we will look into the results

We will be looking into the common cleaning operations good to have in the toolbox while playing with data frames.

• Drop rows having missing values in them and then reset the index of the data frame

• Lower case the column names and then apply a strip function on a column to remove spaces from the values at the beginning and end of the strings

• Apply a function to each value of a column

Sometimes we need to store the data frames after doing processing back into the CSV/JSON files.

   •  Drop rows having missing values in them and then reset the index of the data frame

   •  Delete a column from the data frame

   •  Dump the data frame into the CSV file and a JSON file

We will look into the common uses of Python modules in Data Science(Pandas, NumPy, SciPy, and Matplotlib).

   •  Learn about usage of Pandas module

We will look into the types of Dataframe columns people come across in industry. Numeric columns which contains numbers and then will into their types. Categorical variables and their types.

   •  Understand numerical data

Sometimes a need arises to group similar rows to perform operations on them and get stats out of them.

   •  Perform group by to compute the number of elements in all the groups

   •  Perform group by to compute the average water consumption in every group of animal

Describing features of a column comes in descriptive statistics.

   •  Compute the mean, median and mode of values in a column

   •  Compute sum, standard deviation and range of values in a column

We will look into advanced statistical techniques of computing stats.

   •  Compute geometric mean, harmonic mean and trimmed mean

We are going to look into visualizations. Why they are important? What are the different types?

   •  Understand in detail about visualizations rules

We will see an amazing site showing cool visualizations regarding the happenings in the World over the last 60 years.

   •  Understand visualization with the help of World population dataset

We will look into how we can plot the relationship between variables (scatter plot), look into line plots and the histograms.

   •  Explore the iris dataset and extract the relevant columns out of them

   •  Plot a scatter plot between two columns, plot a line plot of one variable in the data frame

   •  Plot a histogram to  understand its concept in a better way

We will look into box plot and the pie chart of how they can be used to make visualizations of things.

   •  Plot a box plot of a column in data frame which sums up many things in that columns

   •  Make a pie chart for visualizing the utilization of hours of a person in a day

Sometimes it is good to use some online tools which provide many ease to do visualizations for a non-tech person. We will look into RAWGraphs site for making visualizations of the things.

   •  Explore RAWGraphs