Practical Web Scraping Course in Python, Scrapy and Selenium

Tracking and reproducing HTTP requests is an ultimate and primary method of getting data from the Internet. You should always aim to use it, either by replicating your browser's requests or by requesting API access from site owners. In this tutorial, I'll show you how to find the necessary requests and replicate them in a blink of an eye with Postman.

Here is the list of topics we will touch on. First, I’ll explain the difference between the select method and find/find_all methods and why you should prefer the first one more often. Next, we’ll look for use cases when you’d access parents and children elements through specific properties and we’ll review how to get to the text content of tags.

First, let me explain the difference between the select and the find_all methods. Both of them are aiming to capture all elements by some predefined criteria. For example, here’s how you can modify the code from the previous tutorial to use the find_all method and still reach the same result. As you see, the find_all method requires you to define selectors in a pythonic way while the select method allows a more natural, JavaScript-ish way. That’s why I prefer the first one.

The find method does the same thing that the find_all does, but only matches the first element if it exists. It’s not hard to conclude that the same thing can be achieved through the select method and usage of some Python magic.

There are two major use cases I can remember. First, if you have your data packaged into some local HTML or XML file. In that case, you can load it into BS4 and apply its tools to read the markup’s content.

Second, if you’re trying to parse static websites. By static here I mean sites without the usage of javascript for rendering. In other words, if your data is present right away, beautiful soup can be a useful tool. Let us look at an example.

Hi everyone, in this tutorial we will make scraping more dynamic and introduce the selenium library. It is well-known among programmers as a helpful tool in many automation tasks. It can help you simulate user behaviour and therefore pass certain traps along the way of data capturing.

This library helps you navigate through the site using code simulating customers' behaviour. Bunch of stuff you can do with it like:

• Interact with site

• Click on elements

• Drag-N-Drop simulations

• Form filling

• JavaScript execution on page

• DOM search and many more;

In this tutorial I’ll show you how to use the Selenium framework in the most visual possible way. The goal of it is to make you familiar with the tool and those features we would be using in the next lessons. For that purpose, I encourage you to use Jupyter Notebooks as it allows running code step by step with all variables staying in memory. You should be prepared for this if you completed this course's installation tutorial.

This method is helpful for the sake of speeding up your regular scraping tasks. For example, let’s say you’re willing to scrape a site on a daily basis. Instead of logging in each time, you can use a custom scraping profile in your browser to have a session up and running until it is expired by the target site.

Be cautious about this method as some sites would block automation scripts. I’m using such methods typically when I need to perform a single request to the site every once in a while. In later tutorials, I’ll also demonstrate to you how to make your script behave more like a human so it would be possible to decrease the chance of ban.

Services like 2captcha.com can help you. They provide human help for resolving captchas. For this particular tutorial you’d need to create an account there and preferably donate at least 1$ to reproduce the code I’m going to show. Otherwise, this practical exercise would be only theoretical for you.

In this tutorial, we’ll learn how to parse data from JavaScript rendered graphics.

In the first part, you'll see how to create a Python script that captures the reserves of 44 Binance crypto wallets.

Heroku Deployment. Part #2