Web Crawling with Nodejs (H&M, Amazon, LinkedIn, AliExpress)

Name: Web Crawling with Nodejs (H&M, Amazon, LinkedIn, AliExpress)
Rating: 4.4 (121 reviews)

Learn how to create a web crawler using various methods on popular sites like H&M, Amazon, LinkedIn, AliExpress!

Created byStefan Hyltoft

Last updated 11/2020

English

English [Auto],

What you'll learn

Differences between web crawling and web scraping in Nodejs
The 3 main methods to use in web crawling, and when to use what method!
How to get data from sites like H&M and AliExpress easily and fast using their hidden API's
How to build a web crawler for server rendered sites like Amazon to crawl all their products
How to build a Puppeteer based web crawler for a site that requires JavaScript like Linkedin

Course content

5 sections • 38 lectures • 2h 38m total length

What is the web crawling and web scraping and how is it different ?3:42
Learn the basics of web scraping and web crawling, including extracting data from sites into a database, automating hourly checks, and indexing pages across domains.
Legality of web scraping and web crawling4:56
Tools we will be using during development1:18
Discover the development toolkit for building a web crawler: use VS Code or other editors to run and test code, Postman for REST APIs, Chrome DevTools, and methods for scraping.
Methods of web crawling and web scraping2:32
Explore three web crawling methods—from discovering hidden rest apis and simple http requests to puppeteer-powered browser rendering—prioritizing api first, then lightweight requests, then puppeteer, with real-site examples.
Optional: note about deprecation of Request/Request-Promise5:21

Finding hidden API using Chrome Dev Tools6:12
Discover hidden API calls with Chrome DevTools network tab, filter xhr requests, and extract product data (title, price, images) from load more actions, then replicate in Postman.
Testing hidden API inside Postman, and finding other section API endpoints7:34
Identify the hidden rest api with Postman and test the get request to pull new arrivals for women and men, noting data is already formatted for scripting.
Initializing NPM + some info about Nodejs Request and Needle1:45
Creating our HTTP request with needle inside Nodejs2:20
Adding User-Agent header to get past denial in nodejs3:24
Set a user agent header in Node.js requests to avoid access denial, using Needle and browser-like headers. Then fetch all products and save them to a MongoDB database.
Creating MongoDB cluster for saving data4:05
Connecting to MongoDB cluster from Nodejs5:20
Saving data to MongoDB4:53
Getting all products in MongoDB using a loop with offset variable and pagesize7:35
Learn how to fetch all products from an API in Node.js by looping with offset and page size, adding delays, and saving results to MongoDB.

Finding hidden API using Chrome Dev Tools5:24
Learn to uncover hidden APIs with Chrome DevTools by inspecting network traffic and using the mobile page trick to access product data and pagination, then test APIs with Postman.
Making API request from Postman with correct headers4:25
Learn to reproduce mobile API requests in Postman by importing curl requests, identifying essential headers such as user agent, referer, and AMP same origin, and validating responses.
Making API request from Nodejs using Fetch API5:49
Learn how to make a HTTP request using node-fetch and the Fetch API
Getting many items using a for loop and sleep function4:58
Saving AliExpress products to MongoDB3:03

Intro to project1:08
Why are we using HTTP requests and not Puppeteer?4:12
Initializing NPM + installing jest, cheerio and needle npm packages1:26
Writing our reuseable httpRequest module for our testing and crawling2:39
Creating our test HTML file (check resources for URL)5:11
Setting up testing and intro to testing3:26
Writing our first test for our HTML parser4:16
Write the first html parser test for a product page, using before all setup and a read file sync, and verify the title matches the Amazon page.
Getting title from product page and making our test pass5:37
Implement a cheerio-based parser to extract the product title from a page, trim whitespace, and return it in an object, ensuring the test passes; plan to extract the price next.
Getting the price from product page2:50
Making our second test and getting product links from page6:40
Writing out our actual webcrawling in 6 minutes!8:38
Setup so we only crawl only unique product ID's3:07
Adding a new test case for different layout + outtro5:21

Intro to project0:31
Create an automated workflow using Chromium and Puppeteer to log into LinkedIn, visit suggested profiles, build a library of profiles, and extract data.
Initializing project with puppeteer and cheerio packages0:33
Initialize the web crawler project by creating a crawler folder, running npm init, and installing puppeteer and cheerio.
Opening Puppeteer browser and navigating to URL2:43
Login to Linkedin using Puppeteer6:04
Getting profile links on a LinkedIn profile4:47
Log in to LinkedIn, view my profile, reveal related profiles via an extended viewport, extract profile links with a CSS selector using puppeteer and cheerio, and crawl them.
Building web crawler loop for Puppeteer4:32

Requirements

Basic JavaScript

Description

Do you want to build a webcrawler in Nodejs?

In this course you will learn how to build a webcrawler using the newest JavaScript syntax with popular sites like H&M, Amazon, LinkedIn and AliExpress!

You'll learn how to find hidden API's on sites like H&M and AliExpress and see how you can even avoid building a web crawler in the first place, you can save a lot of time this way!

Then I show how to build a web crawler for Amazon the test-driven way, by building out tests for the various product page layouts there is on Amazon.

After that we'll take a look at how to automate login and scraping profiles from LinkedIn using Puppeteer, the automated Chromium browser!

Who this course is for:

Students looking to learn web crawling with Nodejs
Students looking to learn web scraping with Nodejs

Web Crawling with Nodejs (H&M, Amazon, LinkedIn, AliExpress)

What you'll learn

Explore related topics

Course content

Intro to web crawling and web scraping5 lectures • 18min

Getting all products from H&M and saving it to mongodb (Method 1)9 lectures • 43min

AliExpress - getting lots of products and prices using Method 1 (hidden API)5 lectures • 24min

Building a Amazon webcrawler in Nodejs (Method 2, HTTP Requests)13 lectures • 55min

Puppeteer web crawling on LinkedIn6 lectures • 19min

Requirements

Description

Who this course is for: