Learn Web Scraping with NodeJs - The Crash Course

Learn and be great at Web Scraping with NodeJs and tools like: Puppeteer by Google, Request, Cheerio, NightmareJs.

Created byGrohs Fabian

Last updated 1/2024

English

What you'll learn

Create Data Scrapers from Scratch to Finish with NodeJs
Choosing the right tools for Scraping different websites
How to use the Top Scraping tools for NodeJs to your Advantage
How to Automate User Interactions with NodeJs
Build Scrapers with Puppeteer by Google
Build Scrapers with the native Request & Cheerio
Learn how to scrape with NightmareJs

Course content

10 sections • 50 lectures • 7h 8m total length

Introduction2:11
Is this course right for you? Watch this to find out!
Tools & Project Setup3:29
Learn to download and install Node.js and VS Code on Mac or Windows, initialize an npm project, create package.json, and set an entry file index.js to begin coding.
Writing a Simple IMDB Scraper10:31
This is what we learned in this lecture.

Why & When to Choose Scraping3:56
I am talking about when is good to start scraping, what to look for before you start scraping so that you do not get in trouble later on..
The biggest "problem" with scraping2:37
Learn why web scraping breaks as sites change, and how to approach maintenance, debugging, and resilience in scraper workflows to keep data flowing.
Running with Terminal / CMD & Basics on VSCode Debugger6:38
See how to run a Node.js web scraper from the terminal, then use the VS Code debugger to set breakpoints, pause execution, and inspect variables, call stack, and outputs.
The Request library / method5:12
talking about the Request library, example of uses and other informations
The Browser Automation method6:10

01 - How to spoof / add custom request headers6:51
Learn to spoof custom request headers in a nodejs imdb scraper by adding a browser-like user-agent and headers, then test and debug when responses change.
02 - Deal with GZIP Compression enabled websites4:44
Learn to handle gzip compression in web scraping with node.js by enabling gzip in requests and adapting selectors to extract rating values from imdb-like pages.
03 - Selectors & More Parsing19:06
Learn to use selectors and parsing in Node.js to scrape movie data—poster, title, ratings, release date, and genres—using jQuery-like queries with Cheerio and supported by documentation.
04 - Multiple Movies & Saving JSON to File7:59
05 - Exporting data to CSV6:48
In this video we check how you can export data as CSV and save it to a file with NodeJs
06 - Easiest method of getting the proper selectors5:36
Learn web scraping with NodeJs - the crash course demonstrates a fast, browser-driven method to copy selectors for scraping data, explaining when to use it and its reliability limits.
07 - Downloading Images Locally10:21
In this part you will learn how to download images with the request method, piping out the data into an actual file on the system.
08 - Promisifying callback based functions9:12
In this lecture, I am showing you how to work with callback functions and promisify them so that we can wait for specific things to finish.

01 - Installation, introduction and testing4:19
In this lecture we are starting to look at Puppeteer by Google by installing and testing it out for the first time.
02 - Writing a simple automated task9:04
In this part of the lecture we are writing a simple automated task and also checking the documentation on puppeteer.
03 - Generating PDF's, Emulating Phone Views, Getting Title & Url of pages7:36
In this lecture I am showing you:
How to generate a PDF from the browser
Get the Title & Url of the current page
Simulate any phone viewport
04 - Using Puppeteer to login on Instagram11:35
In this lecture, I'm going to show you how easy it is to create a simple login with Puppeteer for Instagram with just a few lines of code.
05 - Making Websites Load Much Faster with a simple trick8:23
I am going to show you a cool trick with Puppeteer that is going to make the website load much faster so that you can save time and bandwidth.
06 - HTTP Basic Authentication3:46
Learn how to handle basic authentication in puppeteer with a single line of code using page.authenticate, passing { username, password } before navigation, and verify success with a test site.
07 - Ignoring HTTPS / SSL Errors & Changing the ViewPort3:40
Learn to configure puppeteer launch options to ignore SSL certificate errors and to set a 1920 by 1080 viewport, ensuring selectors are visible for scraping.
08 - Learning how to use Proxies2:55
Learn how to configure puppeteer on node.js to use a proxy server with the --proxy-server flag, test via an ip endpoint, and note public proxies are risky.
09 - Exposing and working with Custom Functions in Puppeteer9:25
Expose custom functions in puppeteer and run them inside the browser page to safely access title and description, handle selectors, and create reusable helpers like truncate with async and await.

01 - How to login on Twitter10:21
In this part of the course, we are starting to work on the twitter login process with Puppeteer by Google
02 - Refactoring code and improving it7:14
In this part, I am transferring and refactoring code from the Twitter login to a new module so that it is more structured.
03 - How to post Tweets with Puppeteer11:33
Implement a post tweet function for Twitter using Puppeteer in a NodeJs scraper, typing the message via keyboard, waiting for the tweet box, and clicking the post button after login.
04 - Scraping Twitter User Details13:56
In this lesson we start creating a new function to get us the details of any twitter account
05 - Scraping More User Details15:45
In this part we are checking and scraping how to get more details of a user page
06 - Scraping User Tweets14:09
In this part we are going to create a function to scrape the Tweets of any user of Twitter
07 - Scraping More Tweet Details ( Likes, Replies..etc )14:10
In this part we are working on scraping more details for each tweet
08 - Handling The Infinite Loader of the Tweets8:28
In this lesson we are talking about and working on the handler for the infinite loader of the tweets scraper function

01 - Creating the Module Structure6:43
Here we initiate the new project module and prepare everything.
02 - Parsing Product Details11:28
Here we start to process certain details from product pages with Puppeteer on the Amazon website
03 - More Details and Tricks on the Product Details page of Amazon6:59
Here I am going to show you how you can solve certain problems and give you a few tricks on debugging with Puppeteer.

01 - Installation, introduction and testing6:28
Installing the library, going over what it does and testing it out
02 - Talking and testing more about NightmareJs7:15
Master NightmareJS debugging and viewport techniques to build scrapers, compare with Puppeteer, enable detaching DevTools, set custom viewport sizes, and scroll to the page bottom via evaluate.
03 - Scraping Ycombinator News with Vanilla Javascript13:02
04 - Scraping Ycombinator News - More details10:16
Extract key details from Y Combinator News by scraping second table row to obtain the score, author, posting date, and comments while handling missing elements and debugging in the console.
05 - Scraping Ycombinator News - Modulate & Pagination11:40
Modularize a Node.js web scraper for Y Combinator News by building a get_articles function with pagination, exporting a Y Combinator module, and automating page navigation with Nightmare.
06 - Handling Downloads6:06
07 - Injecting Custom Javascript5:23

Requirements

JavaScript Knowledge with ES6 Syntax
Be familiar with CSS / jQuery selectors

Description

Get into the world of Web Scraping and Data Mining with NodeJs. Learn modern methods of scraping with NodeJs - Puppeteer and with direct NodeJs Requests.

Introduce yourself and improve your knowledge on Scraping

Learn Scraping with Puppeteer, NightmareJs or Manual Requests
Build scraper modules for various websites ( Imdb, twitter, instagram..etc )
Learn multiple ways of scraping and when to choose them
Get familiar with the ethics, do's and dont's of Scraping

Enjoy coding and learning Web Scraping with real world examples and real world problem solving while building scrapers with NodeJs.

Web Scraping is a very gray area and not many talk about it or even teach about this. You are going to find valuable scraping informations and techniques that you can directly put to practice for yourself.

I've been working with Data Mining with NodeJs for more than 2 years on dozens of websites and I've learned many ways of creating a scraper and the best practices. All of which you are going to find out and learn in just a few hours in this course.

Overview

I designed this Web Scraping Crash Course to be easily understood by absolute beginners and people who already have some knowledge about the subject.

Complete crash course with all files and code samples, you’ll be able to work alongside with me as we work through each concept and scraper module.

This is not some random tutorial that you usually find on the internet with extremely simple examples. I am showing you everything that you need to think about when starting to build a scraper with NodeJs, while building, problem solving techniques and all you need to know that by the end of the course to be confident and create one for yourself.

Who this course is for:

Complete beginners interested in learning Data Scraping with NodeJs
People who already have some Basic Knowledge but want to Take it to the Next Level

Learn Web Scraping with NodeJs - The Crash Course

What you'll learn

Explore related topics

Course content

Introduction3 lectures • 16min

More Info & Concepts5 lectures • 25min

IMDB Simple Scraper ( Request Method )8 lectures • 1hr 11min

Request Method3 lectures • 33min

Instagram User Simple Scraper ( Request Method )3 lectures • 41min

The Puppeteer Method9 lectures • 1hr 1min

Twitter Scraper with Puppeteer8 lectures • 1hr 36min

Amazon Scraper with Puppeteer3 lectures • 25min

The NightmareJs Method7 lectures • 1hr

The end1 lecture • 1min

Requirements

Description

Who this course is for: