Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Web Scraping with JavaScript & Node.js
Rating: 3.9 out of 5(27 ratings)
249 students

Web Scraping with JavaScript & Node.js

Build your own web scraper. Fetch and process data from whatever website you want!
Created byAdrian Bienias
Last updated 8/2018
English

What you'll learn

  • Using Nodejs
  • Creating and running scripts from packagejson file
  • Using Nodemon for auto refreshing changes in Nodejs code
  • Declaring port number for Nodejs server as environment variable
  • Setting up Nodejs server with Express framework
  • Serving static files with Express framework
  • Fetching and parsing HTML code from urls with JSDOM library
  • Filtering parsed DOM by user names
  • Creating recursive functions
  • Fetching data from URL in front-end JavaScript code with fetch API
  • Inserting custom HTML code into a DOM
  • Creating range sliders with noUiSlider library
  • Passing data from front-end to back-end code with query string
  • Using ES6 syntax
  • Refactoring code

Course content

2 sections • 26 lectures • 8h 22m total length

  • 01. Setting up node.js server23:43

    Learn to set up a node.js server with Express, initialize npm, create routes, and run locally on localhost:3000, while distinguishing public from private directories and using nodemon for auto-restart.

  • 02. Explaining more node.js stuff13:46

    Learn how package-lock.json ensures consistent environments and how to avoid hard-coding ports by using environment variables with process.env, npm scripts, and dev/prod setups.

  • 03. Fetching data from url14:59

    Scrape data from a URL, filter results with a custom advanced search by location and age, and parse the HTML into a document object model for element querying.

  • 04. Parsing html source code11:39

    Explore parsing html source code with jsdom, install and require it, create a dom from data, and use window, document, and selectors to extract elements by id or class.

  • 05. Getting profiles' data20:42

    Inspect search result cards to extract name, photo, photo counter, location, and profile link into a profiles array, and discuss front-end AJAX paging across multiple pages.

  • 06. Passing values with query string13:50

    Learn to pass page, location, sex, and names as query string parameters to build dynamic URLs, read them in code, and return filtered results via the API for the front-end.

  • 07. Serving static files15:49

    Learn to serve static files with express middleware from a public directory, configure API routes, and handle paths for reliable page delivery.

  • 08. Using front end search form19:31

    Implement a front-end search form that prevents default submission, collects input values, and calls a backend API with query parameters via fetch for page-based results.

  • 09. Showing user cards20:34

    Render scraped user data as dynamic cards by iterating an array, constructing DOM elements, and injecting profiles with name, location, and photo into the page.

  • 10. Improving search function20:14

    Improve the search function by building query strings with location, age, and optional sex, handling undefined as empty strings, and control recursive paging with a stop button.

  • 11. Filtering results by names16:32

    Filter search results by names from an input, parsing comma-separated values, trimming, mapping to clean names, and conditionally appending matching profiles to the results.

  • 12. Getting checkboxes' values15:10

    Learn to retrieve all checked checkboxes by name, with or without square brackets, using the checked attribute, and refactor with getSix for API-ready data.

  • 13. Setting age range with noUiSlider17:41

    Learn to set an age range with noUiSlider in a web app, configuring 18–80, starting at 20–50, enabling value displays and steps, and producing a 20–50 string for the backend.

Requirements

  • Modern browser and code editor
  • Basic knowledge of JavaScript

Description

[ NEW RE-RECORDED VERSION! ]

Have you heard about Mark Zuckerberg? He has invented Facebook. But before that, he created site called FaceMash. The goal of that project was to compare 2 girls and choose which one is hotter. They weren't just random girls. They were college students. People knew them and that was the point.

But how Mark got all those girl's photos? It's simple. He fetched them from public search of colleges.

He wrote some script to do that for him, because downloading manually all those pictures would've taken a lot of time.

And this is similar to what we are going to do in this course.

We are going to fetch people's profiles from website called Badoo and filter them by names.

Basically this idea stands even for Google search.

Google is parsing pages from the internet and then filter collected data.

This is exactly what we you're going to learn in this course.

You'll learn how to use JavaScript and Node.js to fetch HTML code from every website you want!

You'll create a project with following features:

  • Getting HTML code from any URL you want with Node.js

  • Parsing HTML code to the DOM with JSDOM library

  • Filtering parsed DOM by user names

  • Passing form values with query strings to Node.js

  • noUiSlider library for nicely looking age range slider

  • Express framework

It's worth mentioning that every lecture has it's own source code files. So if you will somehow stuck you can always compare your code with code from particular lecture.

Now here's the question:

Do you want to effectively learn Node.js with modern JavaScript by creating your own web scraper?

If so, just enroll in this course now.

Who this course is for:

  • Those who want to build own web scraper
  • Beginner Nodejs developers who want to understand how it works
  • JavaScript developers who want to learn new things

Report abuse