The Complete Web Scraping Course with Projects 2019

Explore Node.js, a free, open-source server-side runtime that runs on Windows, macOS, Unix, and Linux, built on Chrome's V8 engine, enabling asynchronous, non-blocking I/O to handle multiple requests.

Install Node.js on Windows by downloading the recommended version, install to the C drive for global access, and check npm version with the command prompt.

Learn how npm, the node package manager, installs, updates, and removes modules locally or globally, with examples for selenium driver and fs, using command prompt or PowerShell.

Choose a code editor to write and debug your web scraping scripts. Sublime Text, Atom, Visual Studio Code, and Nordberg plus plus are free options for organizing and debugging code.

Learn how to automate web scraping to obtain structured data from public websites using Selenium, including collecting, decoding, and saving results to Excel or a database.

Learn to automate web scraping with selenium using a chrome driver to interact with the dom and fetch data, using driver.get, navigate, back, and forward.

Explore how to locate elements with selenium using driver.findElement by id, name, class, and tag name, and master xpath selection with absolute and relative paths.

Apply selenium to perform operations after locating elements, including clicking by id, sending text, and get text, then loop through multiple elements like a fruit list orange, banana, and mango.

Demonstrate basic web scraping syntax with a local example file containing inputs, radios, checkboxes, and lists, using selenium with chrome to load, extract data, and close the browser.

Learn to implement a selenium wait before locating elements to handle slow pages and ajax requests, using a custom function that converts seconds to milliseconds for explicit waiting.

Learn to run a web scraping script, verify output with a console log, and ensure the browser closes by invoking a quit driver with a timed delay.

Automate web form interactions with Selenium by locating inputs by id or name, sending keys, and selecting radio, checkbox, and list options, then log results.

Learn to implement exception handling in web scraping scripts to manage common selenium errors, including element not visible, no such element, invalid selector, and stale element reference.

Take a live screenshot of a web page using a Chrome driver, save the image to an images folder with base64 encoding, and handle success or error in the process.

Explain there are two independent Google Translate scraping projects with different techniques for 2018 and 2019, each with unique codes and decoding exercises to boost your web scraping skills.

Automate the Google Translate workflow using the Nord Jesus script to read languages from translate.google.com and translate an excel file of text into a new output file.

Learn to build a browser driver script to extract language IDs from Google Translate, using Chrome driver and Firefox inspection, and store IDs in an array for later processing.

Learn to build a recursive function that fetches languages related to IDs, using arrays, promises, and incremental indexing to assemble a language list.

Develop a language data export: iterate languages, capture name and id, lowercase names, build a languages object, convert to a json string, and append to a file using fs.

Delete the existing languages file when present, then create a new one to replace it, using a check file existence function with unlink and write.

Run your web scraping script in the background with headless chrome options to avoid opening a browser and collect IDs and languages.

Create a csp file to hold translations from a csv of texts and languages, parse the csv with a csp parser, and prepare the data for scraping translations.

Automate Google Translate scraping with selenium by inputting text into the source box, selecting language, and validating text length with console logs and a recursive language ID approach.

Develop and automate a Google Translate scraping function by fetching data from a browser, selecting languages, clicking translate, extracting translated text, and handling language matching with an Excel file.

Add a function to write translated data into a csv file with columns for text, language, and translated text, including language-not-found handling and escaping.

Enforce a 5000 character limit in text to translate, with a warning when exceeded; implement an if check, clear the source box, and adjust translation delays.

Adjust and implement dynamic delays in web scraping by defining a get source delay function that uses text length and delay factors, then apply source delay in translation workflow.

Test Google Translate workflow by running translations, handling language not found errors, and enforcing a 5000 character limit. Review translator data in Excel and note unicode considerations for the project.

Learn to scrape Google Translate's 2019 UI, build a languages file from the site, and generate a translated data file with original text, language, and translation.

Set up a Google Translate web scrapping project with Chrome driver, creating scripts to fetch language IDs and languages from Afrikaans to Zulu by opening the language box.

Explore automating Google Translate language selection with a recursive function that navigates the language list DOM, uses a hard-coded total, and iterates from the second language onward.

Define a unique language id by parsing class names, splitting on dashes, and extract language names for multilingual pages.

Demonstrates extracting language names from an ID using a custom get language function, navigating language lists with selectors, and enabling headless browser mode for web scraping.

Develop an append-to-array function to assemble a languages map and ids, convert it to json, and write it to a new languages file with the fs module.

Learn to prevent data duplication by checking for and deleting an existing languages.dart file before writing new data. The lecture shows implementing check_file_existence and append_to_file.

Read the translations CSP file with text and language columns, then create a new CSP file containing the translations in German, Filipino, Dutch, Danish, and Irish.

Build an iterative Google Translate scrape function that translates each text item from a translation data file using a recursive loop and send keys to the translator.

Run the GT2K19 script to validate languages in the languages file and extract translated text from Google Translate via a web driver.

Implement robust exception handling for dynamic JavaScript language selection, manage console errors, and address language not found on Google while enforcing a 5000 character translate limit.

Limit translations to 5000 characters by wrapping the translation logic in an if condition, log a console message in the else block, and save results to a CSP file.

Push original text, language, and translated text into a final data array, then write to an Excel file with three columns, handling Google Translate limits and missing languages.

Learn to implement length-based delays in web scraping with gt2k19, replacing fixed wait times with dynamic per-item delays. Account for translation page interactions and exceptions to ensure reliable scrapes.

Automate scraping of a travel insurance site to extract package names and prices. Generate a seven-column csv with trip type, destination, group type, trip days, ages, package name, and price.

Build a second travel insurance scraper project for circle cover, add a dart data file with test cases, and use a languages data file to drive the scraper's inputs.

Define constants and variables for travel insurance, set a global delay factor, and create a generic circle cover file to scrap travel insurance websites.

Create a looping function to process test cases from an array and log progress. Shift through items, check array length, and display messages when no data remains.

Execute a test cases function that opens Chrome, processes cases, and sends data to index.js; derive group type from ages (individual, couple, family) and run a generic runner in index.js.

Develop a run function in the index file, handle callbacks, and save data in CSP format. Explore grouping and destination options for web scraping with circle cover dot com.

Learn to build a JavaScript convert date function that formats dates for the website, correcting month indexing, padding single digits, and joining parts with slashes.

Implement a run function to select the first two fields by using Selenium to choose the policy type and destination, pausing between steps and updating group type and location.

Automate trip duration setup in a web UI: select destination, compute tomorrow's date, and apply start and end dates using the convert date function and send keys.

Learn to automate web form interactions for travel bookings using selenium to set travelers and ages via IDs and for loops, handle dynamic pages, and extract package names and prices.

Compile results by extracting package names and prices from headings and price elements, loop through them, and build a structured reserves array returned via a callback.

Learn to append data to a CSV file using a callback, process reserves with a for loop, and write comma-separated fields to disk.

Debug the web scraping workflow by running loops and test cases, fetching data from the site, and exporting callbacks, while fine-tuning selenium delays for reliability.

Develop a generic web scraping runner to extract insurance package prices and group types from multiple insurers, using modular index files, test cases, and parameterized functions.

Conclude the complete web scraping course with projects by inviting reviews and guiding learners to the library, website, and social channels like YouTube, LinkedIn, Twitter, and Instagram.