Development
Web Development
Node.Js
Learn Web Scraping with NodeJs in 2019 - The Crash Course
4.7 (246 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
1,696 students enrolled

Learn Web Scraping with NodeJs in 2019 - The Crash Course

Learn and be great at Web Scraping with NodeJs and tools like: Puppeteer by Google, Request, Cheerio, NightmareJs.
4.7 (246 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
1,696 students enrolled
Created by Grohs Fabian
Last updated 5/2019
English
Preview this course
Learn Web Scraping with NodeJs in 2019 - The Crash Course
Buy now
30-Day Money-Back Guarantee
This course includes
  • 7 hours on-demand video
  • 32 downloadable resources
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
Training 5 or more people?

Get your team access to 3,500+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • Create Data Scrapers from Scratch to Finish with NodeJs
  • Choosing the right tools for Scraping different websites
  • How to use the Top Scraping tools for NodeJs to your Advantage
  • How to Automate User Interactions with NodeJs
  • Build Scrapers with Puppeteer by Google
  • Build Scrapers with the native Request & Cheerio
  • Learn how to scrape with NightmareJs
Course content
Expand all 50 lectures 07:08:16
+ – Introduction
3 lectures 16:11

Is this course right for you? Watch this to find out!

Introduction
Preview 02:11
Tools & Project Setup
Preview 03:29

This is what we learned in this lecture.

Writing a Simple IMDB Scraper
Preview 10:31
+ – More Info & Concepts
5 lectures 24:33

I am talking about when is good to start scraping, what to look for before you start scraping so that you do not get in trouble later on..

Why & When to Choose Scraping
03:56
The biggest "problem" with scraping
02:37
Running with Terminal / CMD & Basics on VSCode Debugger
06:38

talking about the Request library, example of uses and other informations

The Request library / method
05:12
The Browser Automation method
06:10
+ – IMDB Simple Scraper ( Request Method )
8 lectures 01:10:37
01 - How to spoof / add custom request headers
06:51
02 - Deal with GZIP Compression enabled websites
04:44
03 - Selectors & More Parsing
19:06
04 - Multiple Movies & Saving JSON to File
07:59

In this video we check how you can export data as CSV and save it to a file with NodeJs

05 - Exporting data to CSV
06:48
06 - Easiest method of getting the proper selectors
05:36

In this part you will learn how to download images with the request method, piping out the data into an actual file on the system.

07 - Downloading Images Locally
10:21

In this lecture, I am showing you how to work with callback functions and promisify them so that we can wait for specific things to finish.

08 - Promisifying callback based functions
09:12
+ – Request Method
3 lectures 33:22

In this lecture I am showing you how to set up a proxy server connection with the Request Method.

Using a Proxy Server
04:43

In this video we talk about Status Codes and how we can handle errors in different situations with Request Promise.

Status Codes & Error Handling
10:04
Manual Login with CSRF Protection & Cookie Parsing
18:35
+ – Instagram User Simple Scraper ( Request Method )
3 lectures 41:00
01 - Introduction and inspecting the structures
08:28
02 - Building the scraper
20:24

In this video I am showing you the process on how to parse other data that we get from the Instagram user, more exactly the latest posts details.

03 - Parsing latest posts of an ig user
12:08
+ – The Puppeteer Method
9 lectures 01:00:43

In this lecture we are starting to look at Puppeteer by Google by installing and testing it out for the first time.

01 - Installation, introduction and testing
04:19

In this part of the lecture we are writing a simple automated task and also checking the documentation on puppeteer.

02 - Writing a simple automated task
09:04

In this lecture I am showing you:

  1. How to generate a PDF from the browser

  2. Get the Title & Url of the current page

  3. Simulate any phone viewport

03 - Generating PDF's, Emulating Phone Views, Getting Title & Url of pages
07:36

In this lecture, I'm going to show you how easy it is to create a simple login with Puppeteer for Instagram with just a few lines of code.

04 - Using Puppeteer to login on Instagram
11:35

I am going to show you a cool trick with Puppeteer that is going to make the website load much faster so that you can save time and bandwidth.

05 - Making Websites Load Much Faster with a simple trick
08:23
06 - HTTP Basic Authentication
03:46
07 - Ignoring HTTPS / SSL Errors & Changing the ViewPort
03:40
08 - Learning how to use Proxies
02:55
09 - Exposing and working with Custom Functions in Puppeteer
09:25
+ – Twitter Scraper with Puppeteer
8 lectures 01:35:36

In this part of the course, we are starting to work on the twitter login process with Puppeteer by Google


01 - How to login on Twitter
10:21

In this part, I am transferring and refactoring code from the Twitter login to a new module so that it is more structured.

02 - Refactoring code and improving it
07:14
03 - How to post Tweets with Puppeteer
11:33

In this lesson we start creating a new function to get us the details of any twitter account

04 - Scraping Twitter User Details
13:56

In this part we are checking and scraping how to get more details of a user page

05 - Scraping More User Details
15:45

In this part we are going to create a function to scrape the Tweets of any user of Twitter

06 - Scraping User Tweets
14:09

In this part we are working on scraping more details for each tweet

07 - Scraping More Tweet Details ( Likes, Replies..etc )
14:10

In this lesson we are talking about and working on the handler for the infinite loader of the tweets scraper function

08 - Handling The Infinite Loader of the Tweets
08:28
+ – Amazon Scraper with Puppeteer
3 lectures 25:10

Here we initiate the new project module and prepare everything.

01 - Creating the Module Structure
06:43

Here we start to process certain details from product pages with Puppeteer on the Amazon website

02 - Parsing Product Details
11:28

Here I am going to show you how you can solve certain problems and give you a few tricks on debugging with Puppeteer.

03 - More Details and Tricks on the Product Details page of Amazon
06:59
+ – The NightmareJs Method
7 lectures 01:00:10

Installing the library, going over what it does and testing it out

01 - Installation, introduction and testing
06:28
02 - Talking and testing more about NightmareJs
07:15
03 - Scraping Ycombinator News with Vanilla Javascript
13:02
04 - Scraping Ycombinator News - More details
10:16
05 - Scraping Ycombinator News - Modulate & Pagination
11:40
06 - Handling Downloads
06:06
07 - Injecting Custom Javascript
05:23
+ – The end
1 lecture 00:54
Congratulations!
00:54
Requirements
  • JavaScript Knowledge with ES6 Syntax
  • Be familiar with CSS / jQuery selectors
Description

Get into the world of Web Scraping and Data Mining with NodeJs. Learn modern methods of scraping with NodeJs - Puppeteer and with direct NodeJs Requests.


Introduce yourself and improve your knowledge on Scraping

  • Learn Scraping with Puppeteer, NightmareJs or Manual Requests

  • Build scraper modules for various websites ( Imdb, twitter, instagram..etc )

  • Learn multiple ways of scraping and when to choose them

  • Get familiar with the ethics, do's and dont's of Scraping


Enjoy coding and learning Web Scraping with real world examples and real world problem solving while building scrapers with NodeJs.


Web Scraping is a very gray area and not many talk about it or even teach about this. You are going to find valuable scraping informations and techniques that you can directly put to practice for yourself.

I've been working with Data Mining with NodeJs for more than 2 years on dozens of websites and I've learned many ways of creating a scraper and the best practices. All of which you are going to find out and learn in just a few hours in this course.


Overview

I designed this Web Scraping Crash Course to be easily understood by absolute beginners and people who already have some knowledge about the subject. 

Complete crash course with all files and code samples, you’ll be able to work alongside with me as we work through each concept and scraper module.

This is not some random tutorial that you usually find on the internet with extremely simple examples. I am showing you everything that you need to think about when starting to build a scraper with NodeJs, while building, problem solving techniques and all you need to know  that by the end of the course to be confident and create one for yourself.

Who this course is for:
  • Complete beginners interested in learning Data Scraping with NodeJs
  • People who already have some Basic Knowledge but want to Take it to the Next Level
  •