
Data Wrangling with Python starts with the absolute basics of Python, focusing mainly on data structures, and then quickly jumps into the NumPy and pandas libraries as the fundamental tools for data wrangling. You will learn how, using the Python backend, you can extract and transform data from a diverse array of sources. Then, you will also learn how to handle missinsg or incorrect data, and reformat it based on the requirements from the downstream analytics tool. You will learn about these concepts through real-world examples and datasets.
By the end of this course, you will be confident enough to handle a myriad of sources to extract, clean, transform, and format your data efficiently.
Follow this link to download the code bundle of this course:
https://github.com/TrainingByPackt/Data-Wrangling-with-Python-eLearning
This lesson describes the importance of data wrangling, identifies the important tasks to be performed in data wrangling, and introduces basic Python data structures. Let us cover the following topics:
Importance of Data Wrangling
Sets
Tuples and Strings
Now, let us understand the importance of data wrangling. We’ll also look at lists with some practical examples.
A set, mathematically speaking, is just a collection of well-defined distinct objects. Dictionary is a collection of key-value pairs, where the key can be anything that can be hashed. Let us look at these in detail.
Let us now look at tuples and strings in detail with practical examples.
Let us now summarize our learning from this lesson.
This lesson emphasizes the data structures in Python and the operating system functions that are the foundation of this course.
Let us now look at some of the advanced data structures. In this video, we’ll cover the following sub-topics:
Iterators
Stacks
Lambda expressions
Queues
In this video, we will learn about a few operating system (OS)-level functions in Python. We will concentrate mainly on file-related functions and learn how to open a file, read the data line by line or all at once, and finally how to cleanly close the file we opened. We will apply a few of the techniques we have learned about on a file that we will read to practice our data wrangling skills further.
Let us now summarize our learning from this lesson.
In this lesson, you will learn about the fundamentals of the NumPy, pandas, and matplotlib libraries.
Let us now understand the NumPy arrays. We’ll also look at some of its examples and features along with some demonstrations.
The pandas library is a Python package that provides fast, flexible, and expressive data structures that are designed to make working with relational or labeled data both easy and intuitive. Let us dive deeper into the Pandas DataFrames.
One of the great advantages of using libraries such as NumPy and pandas is that a plethora of built-in statistical and visualization methods are available, for which we don't have to search for and write new code. This video covers the following sub-topics:
Descriptive Statistics
Introduction to Matplotlib Through a Scatter Plot
Definition of Statistical Measures
Random Variables
Probability Distribution
Discrete Distributions
Continuous Distributions
In this video, let us explore a few additional topics related to these libraries, such as how we can bring them together for advanced data generation, analysis, and visualization.
Let us now summarize our learning from this lesson.
In this lesson, we will learn about pandas DataFrames in detail.
In this video, we’ll cover the following sub-topics:
Subsetting the DataFrame
The unique Function
Conditional Selection and Boolean Filtering
Setting and Resetting the Index
The GroupBy Method
In this video, we’ll cover the following sub-topics:
Outlier detection
Missing Values in Pandas
Filling and dropping missing Values in Pandas
Outlier Detection Using a Simple Statistical Test
Merging and joining tables or datasets are highly common operations in the day-to-day job of a data wrangling professional. Through this video, let us understand how the pandas library offers nice and intuitive built-in methods to perform various types of JOIN queries involving multiple DataFrame objects.
In this video, we will discuss some small utility functions that are offered by pandas so that we can work efficiently with DataFrames.
Let us now summarize our learning from this lesson.
In this lesson, you will be exposed to real-life data wrangling techniques, as applied to web scraping.
In this video, we will go through various data sources and how they can be imported into pandas DataFrames, thus imbuing wrangling professionals with extremely valuable data ingestion knowledge.
In this video, we will cover the reading and parsing of web pages, but we do not request them from a live website. Instead, we read them from disk.
Let us now summarize our learning from this lesson.
In this lesson, you will learn about data issues that happen in real-life. You will also learn how to solve these issues.
In this video, we will deep dive into the heart of list comprehension. We will explore the power of this amazing tool further. We will investigate another close relative of list comprehension called generators, and also work with zip and its related functions and methods.
In this video, we will format a given dataset.
Some basic techniques that are commonplace to flag and filter outliers in real-world data for day-to-day work are covered in this lesson.
Let us now summarize our learning from this lesson.
In this lesson, you will learn how to gather data from web pages, XML files, and APIs.
In this video, we will build a real-life web scraper using all of the techniques that we have learned so far.
XML, or Extensible Markup Language, is a web markup language that's similar to HTML but with significant flexibility built in, such as the ability to define your own tags. Through this video, we’ll explore how to read data from XML.
Fundamentally, an API or Application Programming Interface is some kind of interface to a computing resource, which has a set of exposed methods that allow a programmer to access particular data or internal features of that resource. Let us now understand how to read data from an API.
Regular expressions or RegEx are used to identify whether a pattern exists in a given sequence of characters a (string) or not. This video gives us a deep understanding of the fundamentals of RegEx.
Let us now summarize our learning from this lesson.
This lesson explains the concepts of databases, including their creation, manipulation, and control, and transforming tables into pandas DataFrames.
An RDBMS is a piece of software that manages data on physical hard disks. Structured Query Language or SQL is a domain-specific language that is widely used to define, insert, manipulate, and retrieve data from the databases. This videos gives us a refresher of RDBMS and SQL.
In this video, we will focus on how to write some basic SQL commands, as well as how to connect to a database from Python and use it effectively within Python.
Let us now summarize our learning from this lesson.
For data to be useful and meaningful, it must be curated and refined. Data Wrangling with Python teaches you the core ideas behind these processes and equips you with knowledge of the most popular tools and techniques in the domain.
The course starts with the absolute basics of Python, focusing mainly on data structures. It then delves into the fundamental tools of data wrangling like NumPy and Pandas libraries. You'll explore useful insights into why you should stay away from traditional ways of data cleaning, as done in other languages, and take advantage of the specialized pre-built routines in Python. This combination of Python tips and tricks will also demonstrate how to use the same Python backend and extract/transform data from an array of sources including the Internet, large database vaults, and Excel financial tables. To help you prepare for more challenging scenarios, you'll cover how to handle missing or wrong data, and reformat it based on the requirements from the downstream analytics tool. The course will further help you grasp concepts through real-world examples and datasets.
By the end of this course, you will be confident in using a diverse array of sources to extract, clean, transform, and format your data efficiently.
About the Author
Samik Sen is currently working with R on Machine Learning. He has done his Ph.D. in Theoretical Physics. He has Tutored Classes for High-Performance Computing postgraduates and Lecturer at International Conferences. He has experience of using Perl on data, producing plots with gnuplot for visualization and latex to produce reports. He, then, moved to finance/football and online education with videos.
Dr. Tirthajyoti Sarkar works as a senior principal engineer in the semiconductor technology domain, where he applies cutting-edge data science/machine learning techniques for design automation and predictive analytics. He writes regularly about Python programming and data science topics. He holds a Ph.D. from the University of Illinois and certifications in Artificial Intelligence and Machine learning from Stanford and MIT.
Shubhadeep Roychowdhury works as a senior software engineer at a Paris-based cybersecurity startup, where he is applying the state-of-the-art computer vision and data engineering algorithms and tools to develop cutting-edge products. He often writes about algorithm implementation in Python and similar topics. He holds a master's degree in computer science from West Bengal University Of Technology and certifications in machine learning from Stanford.