Learn to scrape any website with R

Name: Learn to scrape any website with R
Rating: 4.3 (76 reviews)

Get data from the web directly into R

Last updated 7/2020

English

What you'll learn

6 sections • 30 lectures • 2h 48m total length

Introduction3:07
Prerequisites1:15
Scrape a table from Wikipedia in 20 lines of code - part 17:20
Scrape a table from Wikipedia in 20 lines of code - part 27:30
Learn to scrape a Wikipedia table in 20 lines of code, clean and reshape data with R by selecting columns, renaming them, removing footnotes and nonnumeric values for analysis.

3 HTML basics: tags, classes and id's5:55
Explore how HTML uses tags, classes, and IDs to structure and style pages for scraping. Learn to select headings and content with tag, class, and id to extract exact data.
Introduction to section: Scrape Trustpilot2:01
Read in HTML and create a list of reviews3:59
Learn to read html with R, extract a list of reviews by selecting elements with the review class, and loop through each item to assemble review data for consumers.
Extract the content of the reviews14:24
Extract and structure each review by retrieving the id, title, text, and star rating, then parse the timestamp from embedded JSON for analysis.
Extract the content about the reviewer5:32
Create a function and extract review data to a data frame5:11
Loop over all pages and collect all reviews10:26
Clean up the scraped data5:50
Learn to clean scraped data by converting reviews and stars to numbers, formatting timestamps, and preparing a ready-to-analyze data frame for any web page.

First attempt - and fail3:34
Explained: Server-side vs Client-side rendering4:11
Scraping a client-side rendered webpage9:12
Extract a list of products3:30
Learn to extract a list of products from a webpage using R by locating the product list container and iterating through product items, then gathering data across pages.
Extract the details for each product8:18
Create a function and extract product data to a data frame2:28
Prepare to loop over all product pages6:44
Run loop and save HTML5:37
Parse the saved HTML to a data frame4:29
Clean the saved data - part 19:02
Learn to clean and standardize saved data by extracting numeric volume from product data and removing units. Create a new volume column while handling lists with map.
Clean the saved data - part 28:20
Tip: Save the page yourself4:58

In this course you will learn:

If you have ever wanted to collect:

Then this course is for you!

At the end of the course you will be able to:

Extract data from structured tables on the internet
Collect text data from a website (like reviews) and structure it in a tidy data set
Collect a diverse set of data points from a website (like products) and structure it in a tidy data set
Collect structured data from an API - ready for analysis
Scrape websites that are normally very hard to extract data from.