Web Crawling with Nodejs (H&M, Amazon, LinkedIn, AliExpress)

Name: Web Crawling with Nodejs (H&M, Amazon, LinkedIn, AliExpress)
Rating: 4.4 (121 reviews)

Learn how to create a web crawler using various methods on popular sites like H&M, Amazon, LinkedIn, AliExpress!

Created byStefan Hyltoft

Last updated 11/2020

English

What you'll learn

Differences between web crawling and web scraping in Nodejs
The 3 main methods to use in web crawling, and when to use what method!
How to get data from sites like H&M and AliExpress easily and fast using their hidden API's
How to build a web crawler for server rendered sites like Amazon to crawl all their products
How to build a Puppeteer based web crawler for a site that requires JavaScript like Linkedin

Course content

5 sections • 38 lectures • 2h 38m total length

What is the web crawling and web scraping and how is it different ?3:42
Legality of web scraping and web crawling4:56
Tools we will be using during development1:18
Methods of web crawling and web scraping2:32
Optional: note about deprecation of Request/Request-Promise5:21

Finding hidden API using Chrome Dev Tools6:12
Testing hidden API inside Postman, and finding other section API endpoints7:34
Initializing NPM + some info about Nodejs Request and Needle1:45
Creating our HTTP request with needle inside Nodejs2:20
Adding User-Agent header to get past denial in nodejs3:24
Creating MongoDB cluster for saving data4:05
Connecting to MongoDB cluster from Nodejs5:20
Saving data to MongoDB4:53
Getting all products in MongoDB using a loop with offset variable and pagesize7:35

Intro to project1:08
Why are we using HTTP requests and not Puppeteer?4:12
Initializing NPM + installing jest, cheerio and needle npm packages1:26
Writing our reuseable httpRequest module for our testing and crawling2:39
Creating our test HTML file (check resources for URL)5:11
Setting up testing and intro to testing3:26
Writing our first test for our HTML parser4:16
Getting title from product page and making our test pass5:37
Getting the price from product page2:50
Making our second test and getting product links from page6:40
Writing out our actual webcrawling in 6 minutes!8:38
Setup so we only crawl only unique product ID's3:07
Adding a new test case for different layout + outtro5:21

Requirements

Basic JavaScript

Description

Do you want to build a webcrawler in Nodejs?

In this course you will learn how to build a webcrawler using the newest JavaScript syntax with popular sites like H&M, Amazon, LinkedIn and AliExpress!

You'll learn how to find hidden API's on sites like H&M and AliExpress and see how you can even avoid building a web crawler in the first place, you can save a lot of time this way!

Then I show how to build a web crawler for Amazon the test-driven way, by building out tests for the various product page layouts there is on Amazon.

After that we'll take a look at how to automate login and scraping profiles from LinkedIn using Puppeteer, the automated Chromium browser!

Who this course is for:

Students looking to learn web crawling with Nodejs
Students looking to learn web scraping with Nodejs

Web Crawling with Nodejs (H&M, Amazon, LinkedIn, AliExpress)

What you'll learn

Explore related topics

Course content

Intro to web crawling and web scraping5 lectures • 18min

Getting all products from H&M and saving it to mongodb (Method 1)9 lectures • 43min

AliExpress - getting lots of products and prices using Method 1 (hidden API)5 lectures • 24min

Building a Amazon webcrawler in Nodejs (Method 2, HTTP Requests)13 lectures • 55min

Puppeteer web crawling on LinkedIn6 lectures • 19min

Requirements

Description

Who this course is for: