Web Crawling with Nodejs (H&M, Amazon, LinkedIn, AliExpress)

Discover hidden API calls with Chrome DevTools network tab, filter xhr requests, and extract product data (title, price, images) from load more actions, then replicate in Postman.

Identify the hidden rest api with Postman and test the get request to pull new arrivals for women and men, noting data is already formatted for scripting.

Initialize npm in an empty hm scraping folder and set up an index. Use the needle package for HP requests and note the deprecated status of the request package.

Set a user agent header in Node.js requests to avoid access denial, using Needle and browser-like headers. Then fetch all products and save them to a MongoDB database.

Set up a free cloud MongoDB cluster, create a project and user, whitelist your IP, and obtain a connection string to support your Node.js web crawling workflow.

Learn how to fetch all products from an API in Node.js by looping with offset and page size, adding delays, and saving results to MongoDB.

Learn to uncover hidden APIs with Chrome DevTools by inspecting network traffic and using the mobile page trick to access product data and pagination, then test APIs with Postman.

Learn to reproduce mobile API requests in Postman by importing curl requests, identifying essential headers such as user agent, referer, and AMP same origin, and validating responses.

Build a test-driven web crawler for Amazon.com to extract product titles, prices, and details from product pages and follow links for complete coverage.

Explain why we use http requests over puppeteer for scraping Amazon, focusing on server-rendered pages. Prefer simple http requests when hidden rest APIs exist, and reserve puppeteer for javascript-heavy rendering.

Develop a reusable http request module with needle to fetch and save html for testing and crawling across amazon product pages, enabling test files for different layouts.

Set up a test folder and files to show how testing works, using describe and test to verify a parser in the web crawler and npm run test.

Write the first html parser test for a product page, using before all setup and a read file sync, and verify the title matches the Amazon page.

Implement a cheerio-based parser to extract the product title from a page, trim whitespace, and return it in an object, ensuring the test passes; plan to extract the price next.

Write a simple Node.js web crawler that visits pages, extracts title, price, and product data, and tracks visited and to-visit links with a while loop and sleep.

Learn to crawl only unique product IDs by cleaning and normalizing links, and extracting IDs with a first 14 characters slice to keep results clean.

Add a new test case for different product page layouts and price extraction, and build a modular crawler with a dedicated parser, preparing for puppeteer web crawling.

Create an automated workflow using Chromium and Puppeteer to log into LinkedIn, visit suggested profiles, build a library of profiles, and extract data.

Initialize the web crawler project by creating a crawler folder, running npm init, and installing puppeteer and cheerio.

Log in to LinkedIn using Puppeteer to fill the username and password, click sign in, and crawl related profiles after adjusting the viewport.

Log in to LinkedIn, view my profile, reveal related profiles via an extended viewport, extract profile links with a CSS selector using puppeteer and cheerio, and crawl them.

Build a puppeteer web crawler loop that visits LinkedIn profiles, manages links to visit and visited links, and scrapes profile data with controlled delays.