Data Acquisition and Manipulation with Python

Name: Data Acquisition and Manipulation with Python
Rating: 4.5 (5 reviews)

Acquire and analyse data in different formats with the help of Python data analysis tools

Created byPackt Publishing

Last updated 10/2017

English

What you'll learn

See how to acquire and save different formats of data
Find out how to connect to a database and add information to it
Combine and merge data sets, and manipulate strings using Python
Aggregate your data and employ group-wise operations and transformations
Know how to download and read a web page using the BeautifulSoup package
Extract useful information from websites using Selenium in Python
Program and employ a Scrapy spider for successful web scraping

Course content

6 sections • 25 lectures • 2h 39m total length

The Course Overview5:37
This video gives an overview of entire course.
Getting Data in Its Different Formats9:05
In this video, we will see how we load data into Python. Here we see, how to load data into a Pandas DataFrame from various sources, including CSV, Excel, XML, JSON, and via a web API.
Connecting to a Database4:47
In this video, we will see how we can connect to a (MySQL) database. Here we set up such connection.
Adding Information to a Database3:31
Once we have a database, we will see how we add data to it. Here, we create a table in Python with SQL commands, then use a Panda’s dataframe to add data to the table in the database.
Query a Database3:35
In this, we will see how we extract data from a database. Here we use SQL and Pandas to load a table’s data into a DataFrame.

Combining and Merging Datasets5:25
Different datasets may contain different information that we may want together. Here in this video, we use joins to combine these datasets together.
Reshaping Data8:46
Datasets may not come in appropriate formats or shape. Here, this video demonstrate how to manipulate a dataset’s shape, altering what is represented in rows and what is in columns.
Data Transformation8:48
Columns in a dataset may not represent data in a preferred manner. Here in this video we look at different data transformation techniques to remedy this.
String Manipulation9:12
Strings may contain useful data. Here we will show how to manipulate string contents and extract data from them.

Grouping Basics6:00
In this video, we will show how we can form and use groups. We will use the DataFrame method groupby() to create groups, then will do the group operations you want.
Data Aggregation6:22
In this video, we will learn how we get numerical summaries of groups. We will use data aggregation methods, such as Series/DataFrame methods, or agg().
GroupWise Operations and Transformations7:12
In this video, we will see how we can perform operations at group levels. We will use split-apply-combine procedures.
Pivot Tables and Cross Tabulation9:21
In this video, we will show how we can perform cross tabulation or create pivot tables. We will use crosstab() or pivot_table().

Exploring Web Scraping8:49
In this video, we will show how we get messy data from web pages. Also how to write a web scraper.
Scraping the Web Safely7:55
In this video, we will see how we can practice safe web scraping. Also we will highlight some issues for you to consider.
Let’s Have a BeautifulSoup8:19
In this video, we will see how we can get data from a simple HTML document. Also how to use BeautifulSoup.
Downloading and Reading a Webpage4:40
This video will show how we actually use BeautifulSoup. Here this video show an example.
Extracting Useful Information6:04
In this video, we will see how we can create a dataset spread across multiple web pages. We crawl the web with Python and BeautifulSoup.

Making Use of Selenium4:43
This video explains what a Selenium is. It is a package for automating a Web browser.
Navigating the Web with Selenium6:48
This video explains how Selenium navigates the web. It does so much like a human would.
Extracting Information with Selenium5:57
In this video, we will see how we get data from a webpage’s dynamic content. Load the webpage with Selenium, let the dynamic content activate, then read the page.

Diving into Scrapy8:21
In this video, we will show what a Scrapy is. It is a software suite built on Python for Web crawling and scraping.
Creating a Scrapy Spider2:42
In this video, we will see how do we create a Scrapy spider? Scrapy provides an automated process for creating projects and their spiders.
Programming a Spider3:51
In this video, we wil show how we make Scrapy spiders collect data. We program them, focusing on their parse() method.
Deploying a Spider3:29
Once we have a Scrapy spider programmed, we will see how we use it. We run it with the crawl command.

Requirements

• What will the customer need to install (core software) to follow along? Anaconda, Python, MySQL, Chrome or Firefox
• What Plugins will the customer need? Install packages as needed
• What versions of software should the customer use? Relatively recent, but not version sensitive
• Any special hardware needed? No
• Can you point to instruction to install anything that we will need to install? (Can be websites, other videos, written instructions) I detail installation in Unpacking NumPy and Pandas, an earlier course. Otherwise, check Chrome/Firefox's site, continuum .io for Anaconda, oracle .com for MySQL
• Anything that you would like the customers to watch out for like a precautionary measure or a tip? DO NOT SCRAPE THE WEB TOO FAST!!! Also, practice safe web practices.
• How are the Exercise Files laid out? Code Samples/ Starting State of the Examples/ Finished State of the Examples/ Finished State per video/ Finished State per Chapter/ Finished State per course Exercises are often linked within a section. Many need data files that I have provided. A set of exercises become a guided project that allow viewers to do tasks similar to those shown in the videos. They simply give viewers instructions on what they need to do to complete the exercise.
• What level user would be best served by this course? For example: "Beginner Django User but Advanced Web Designer"
• This course has been tested on the following system configuration: ● OS: Windows 10 ● Processor: AMD 64 ● Memory: 12GB ● Hard Disk Space: 300GB ● Video Card: NVidia 560

Description

Python, a multi-paradigm programming language, has become the language of choice for data scientists for data analysis, visualization, and machine learning.

In this course, you’ll start by learning how to acquire data from the web in its already “clean” format, such as in a .csv file, or a database. You’ll then learn to transform this data so it’s in its most useful format for analysis. After that, you’ll dive into data aggregation and grouping, where you’ll learn to group similar data for easier analysis purposes. From there, you’ll be shown different methods of web scraping using Python. Finally, you’ll learn to extract large amounts of data using BeautifulSoup, as well as work with Selenium and Scrapy.

About the author

Curtis Miller is Associate Instructor at the University of Utah, and an MSTAT student. He is currently involved in research on data analysis from statistical and computer science perspectives. Curtis has published research on policy and economic issues.

Who this course is for:

If you are a budding Data Analyst and you want to learn how to acquire, manage, and manipulate large amounts of data from the web using the Python programming language, then this is the perfect video tutorial for you.

Data Acquisition and Manipulation with Python

What you'll learn

Explore related topics

Course content

Getting and Saving Data5 lectures • 27min

Data Wrangling4 lectures • 32min

Data Aggregation/Grouping4 lectures • 29min

Web Scraping with BeautifulSoup5 lectures • 36min

Web Scraping with Selenium3 lectures • 17min

Web Scraping with Scrapy4 lectures • 18min

Requirements

Description

Who this course is for: