Web Scraping Mastery with Python 2023

Name: Web Scraping Mastery with Python 2023
Rating: 4.5 (27 reviews)

Become an expert in web scraping, finding hidden api's, reversing them and bypassing web automation/bot detection.

Highest Rated

Created byMarco Capuano

Last updated 6/2023

English

What you'll learn

Learn about the web scraping fundamentals
Learn about the top 3 web scraping methods
Learn to analyze websites and choose the optimal web scraping method
Learn to find hidden API's and reverse them
Learn to perform request and emulation based web scraping
Learn to parse HTML & JSON
Learn to optimize your requests and create clean fingerprints
Learn to bypass many anti-bot detections with selenium emulation (Advanced & unique)
Learn web-automation with selenium
Learn to scrape e-commerce stores and review websites

Course content

4 sections • 14 lectures • 4h 40m total length

Web Scraping process14:53
In this lecture you will learn about the Web Scraping process overall. Its essential to decide which scraping method we want to apply in the future, which options we have and challenges we might encounter.
Using chrome dev tools to analyze websites14:29
In this course you will learn to analyze websites with the chrome dev tools and decide which scraping method works best. We are taking a look at two different e-commerce stores and in addition find hidden api's and request based scrape websites which seem to be only scrapeable with emulation, which sets you atleast 100 steps ahead of other web scrapers.
Choosing the appropriate scraping method
HTML & JSON parsing27:12
In this lecture we take a quick look at how we can extract data from html code or a json response.
The Art of Parsing (New additional content)27:33
This is an additional lecture about Parsing with Python. I''ve created this additional content due too some messages i've recieved, im explaining the logic of parsing and how parser (like Beautiful Soup) work in more detail and then reexplain the content of the last lecture! If you are already skilled at parsing maybe take a look at the first few minutes and then skip this lecture. Enjoy!
Parsing practice quizz

Web Request explained6:59
In this video we will take a quick look at web requests (HTTP).
[Project] Request based web scraping product listings15:25
In this lecture we web scrape our first website!
We are web scraping 1600+ product titles and prices of an e-commerce store.
[Project] Advanced request based web scraping27:59
This lecture is an advanced version of the last project!

We will find and reverse an API of the same target website, which is a good example of the multiple ways a website can be scraped.
The 4 Layers of web security you will encounter13:02
In this lecture we will take a look at the 4 Web Security layers you will encounter and a quick introduction at how we could potentially bypass each of them.
[Project] Trustpilot review scraping bypassing security layer 1 & 223:55
In this lecture we are scraping trustpilot reviews request based. In addition we are reversing a hidden api, use proxies and custom headers.

Web emulation explained7:01
In this video you will learn about the difference of browser automation and emulation and something about the features and upsides of emulation.
First emulation with selenium & chromium3:51
In this lecture we will perform our first web emulation with selenium and chromium driver.
[Project] Web automation with selenium @ google.com18:15
In this lecture we get to know selenium, and perform a google search query fully automated.
Spoofing the emulation & creating a reuseable driver creation class29:51
In this lecture we are looking at how to make our chromedriver undetectable and seem like a legit browser, in addition we will create a class which we can use in future in any web scraping projects.
[Project] Scraping Google places reviews (Emulation + API Reversal -> Requests)35:01
In this lecture we will emulation based scrape google places.

Requirements

Basic knowledge in Python
Internet Access
Hardware with minimum of 4GB RAM

Description

Web Scraping has become an art in todays online world, more and more websites learn to block automated traffic or generate their data dynamically. Common web scraping techniques are deprecated and tutorials outdated.

In addition, the first thing you will see when you google web scraping, are proxy or scraping service providers, which earn alot of money by providing scraping services. Those services use certain tricks to provide scraping services for important websites like amazon, google etc., but they wont share those tricks (I do in this course). They claim to use emulation or simmilar techniques to scrape their targets, but in reality they use much cheaper and more efficient ways of scraping the desired data.

In this course you will learn the web scraping fundamentals within an hour and then proceed with learning those secret tricks, big web scraping providers use.

What can you expect?

We will take a look at the web scraping fundamentals and exercise choosing the optimal scraping solution...

We will build projects to scrape review data of trustpilot, amazon, google places and so on...

We will solve real world practice exercises...

You will have advanced knowledge about anti-bot detections and security mechanisms of websites...

You will learn about advanced web automation...

You will learn about customizing your fingerprint and bypass automation detections...

You will learn to reverse complex API's and practice it with real world examples...

and so much more...

Who this course is for:

Anyone who wants to scrape data from any website
Skilled developers who want to obtain advanced knowledge in Web Scraping
Entrepreneurs who want to compete in the global Web Scraping & Web-automation market
Anyone who wants to stop spending hours manually copying data off a website, and wants to automate this task
Anyone who wants to scrape Google, Amazon etc.

Web Scraping Mastery with Python 2023

What you'll learn

Explore related topics

Course content

Web Scraping Fundamentals4 lectures • 1hr 24min

Request based web scraping5 lectures • 1hr 27min

Emulation based web scraping & web automation5 lectures • 1hr 34min

Assignment0

Requirements

Description

Who this course is for: