Web Scraping with Python: BeautifulSoup, Requests & Selenium
4.1 (773 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
5,440 students enrolled

Web Scraping with Python: BeautifulSoup, Requests & Selenium

Web Scraping and Crawling with Python: Beautiful Soup, Requests & Selenium
4.1 (773 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
5,440 students enrolled
Last updated 12/2018
English, Italian [Auto]
Current price: $69.99 Original price: $99.99 Discount: 30% off
5 hours left at this price!
30-Day Money-Back Guarantee
This course includes
  • 8 hours on-demand video
  • 10 articles
  • 19 downloadable resources
  • 1 coding exercise
  • Full lifetime access
  • Access on mobile and TV
  • Assignments
  • Certificate of Completion
Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • Python Refresher: Review of Data Structures, Conditionals, File Handling
  • How Websites are Hosted on Servers; Basic Calls to Server (GET, POST Methods)
  • Web Scraping with Python Beautiful Soup and Requests
  • Using Selenium to handle JavaScript and AJAX
  • Diverse Web Scraping Exercises
  • Source codes (*.py files) for all Exercises can be downloaded
  • Q&A board to send your questions and get them answered quickly
Course content
Expand all 72 lectures 07:55:54
+ Web Scraping Course Overview
1 lecture 04:22

- [Instructor] Welcome everybody to the course. I'm really glad that all of you are here. Thank you again for checking out this course so, I'll start off explaining what web scraping is, as this course is all about web scraping. So, in simple and brief terms, web scraping is extracting data from the internet. Why do we need to do that? That's a valid question.

Why do we need to scrape data from the internet?

We live in an age where everything is being uploaded on the internet, you know, daily terabytes of data is being uploaded on the internet. And with such an amount of data, you could perform analysis and you could actually improve a lot of things. Like, if you were a businessman and if you're launching your product, you could learn more about the market, e.g. what do people like? How will they respond to your product launch, or how can you improve your product. 

All of this information is on the internet so you need some tool, some way to actually get that information out of the internet. One way would be to hire hundreds of persons to do this stuff, or the other way, the smart way would be that you write a computer program which does this for you, and conserves your resources. So, this is what web scraping is about.

The next question is what is this course about?

This course is about how can we access webpages programmatically? We'll be using Python for this course, so this course will teach us how we can parse a webpage, extract required data from that webpage.

Let's say you want to get some photos from some webpage, how are you going to do that? Let's say there are thousands of photos. Don't tell me that you're going to download each of them individually? I mean that is going to take a lot of time, so we will learn how to scrape a web page and extract our required data from our web pages.

Python has this module, which is known as BeautifulSoup, this is a parser for parsing web pages. We'll learn this too in this course. So in this course, we'll also learn how we can interact differently with our web pages and how we can move around and play around with them, you know?

We'll also learn that and you could say that it is an introduction to automation, you know, automating different stuff on the internet. You can say that in simple terms. So we'll be using Selenium for interacting with web pages in this course.

So to sum it up, to sum the answer to this question, I can say that briefly it is about how we extract data from any kind of web page using a programming language, in our case, it is going to be Python.

What will you learn by the end of this course?

At the end of this course, you will have a deep understanding of how web sites and servers function, like how are web sites hosted and how do they send requests to servers? What are servers? What are websites? And how does this communication take place? So you'll have a deeper understanding of how this works, how all of this works.

Then, you'll learn different web scraping and data extraction techniques which are being used worldwide, like the best practices, the worst practices. Things you have to worry, things you have to keep in mind while you are scraping data, and how can you do this in an efficient manner. You will learn this by the end of this course. 

You'll also learn how to handle a lot of data and how we can actually parse that data , and get it in the required format we want.

So I really look forward to teaching you this course. I hope that you're excited about this. I think let's get to our first tutorial, so I'll see you soon in the next video. Thank you!

Preview 04:22
+ Python Refresher: Data Structures (Optional)
11 lectures 01:08:00

How to use lists in Python. Revision of Python Lists and commonly used list functions.


How to use dictionaries in Python. Revision of Python Dictionaries and commonly used dict functions.


How to use tuples in Python. Revision of Python Tuples and commonly used tuple functions.


How to write simple list comprehensions. Python supports a concept called "list comprehensions". It can be used to construct lists in a very natural, easy way, like a mathematician is used to do. List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of

some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.

For example, assume you want to create a list of squares, instead of writing a regular for loop, you can use a list comprehension like:

squares = [x**2 for x in range(10)]

List Comprehensions - Part 1

Writing complex list comprehensions.

List Comprehensions - Part 2

Using if else conditions while writing list comprehensions.

Inline - if else and List Comprehensions
Installing xlrd and XlsxWriter​ to Read/Write to Excel Files

Introduction to the Python xlsxwriter module which can be used to create Excel files and write into them. This video discusses on creating and writing data to Excel files using xlsxwriter.

Preview 10:02

Introduction to Python xlrd module which is used to read data from Excel files. This video discusses how we can read Excel files using xlrd.

Reading from Excel Files
Python Editor & Other Software
Exercise #1: YOU: Web Scraping Expert
+ How Servers Work
2 lectures 03:54

What servers are and how they work in terms for hosting websites.

How Websites are Hosted

What is HTML? Basics of HTML

Preview 02:22
+ BeautifulSoup Warm-up Exercise
1 lecture 01:38
BeautifulSoup Solved Exercise
Simple Scraper
1 question
+ Installing Required Python Packages
1 lecture 03:19
Installing Required Python Packages
+ Introduction to Requests Python Library
3 lectures 15:56
Introduction to Python module "Requests" which can send requests to websites servers and receive responses.
Requests Get Method

How to fake to a website’s server that your Python code is a browser.

User Agent
Installing fake_useragent Package
+ Introduction to Beautiful Soup Python Library
4 lectures 18:18

Introduction to Beautiful Soup - a Python module used for parsing HTML.

Preview 04:44

Understanding the HTML parse tree which is created by Beautiful Soup.

Web Scraping with Beautiful Soup - Overview P.2

How we can access tags in the HTML parse tree generated by Beautiful Soup.

Accessing Tags

Concept of navigable strings in Beautiful Soup and how we can access them.

Navigable Strings
+ Navigating with Beautiful Soup - Going Down
3 lectures 18:54

How we can navigate our parse tree using tag names.

Navigating through Tag Names

How we can access a tag’s direct children.

Contents and Children Methods

How we can access a tag’s all descendants.

Descendants Method
+ Navigating with Beautiful Soup - Going Up
2 lectures 11:42

How to get to the parent tag of a tag.

Parent Method

How to get all the ancestors of a tag.

Parents Method
+ Navigating with Beautiful Soup - Going Sideways
3 lectures 13:16

How to access the next sibling element of a tag in our parse tree with BeautifulSoup.


How to access the previous sibling tag of a tag in the parse tree with BeautifulSoup.


How to access all the next sibling tags and all the previous sibling tags of a tag in our parse tree with BeautifulSoup.

next_siblings & previous_siblings
  • Some prior programming experience in Python (e.g. Data Structures and OOP) will help. The course includes a full Python refresher section.
  • Complete beginners may wish to take a beginner Python course first, and then transition to this course afterwards.
  • This course adopts a step-by-step approach and requires you to open a Python editor, download available *.py code files, and start applying the provided examples and exercises.
  • Python 3: Codes of this course are tested on Python 3. It is up to you to adapt them if you want to run them in Python 2.

Web Scraping (also termed Screen Scraping, Web Data Extraction, Web Harvesting, etc.) is a technique for extracting large amounts of data from websites  and save the the extracted data to a local file or to a database.

In this course, you will learn how to perform web scraping using Python 3 and the Beautiful Soup, a free open-source library written in Python for parsing HTML.

We will use lxml, which is an extensive library for parsing XML and HTML documents very quickly; it can even handle messed up tags. We will also be using the Requests module instead of the already built-in urllib2 module due to improvements in speed and readability.

Finally, we will use Selenium alongside Beautiful Soup to crawl AJAX & JavaScript driven pages.

The course cover the following topics: accessing web pages programmatically; scraping web pages to extract the required data using Beautiful Soup to parse web pages; interacting with web pages to do different things with them programmatically; and using Selenium for web scraping and when we need it.

By the end of this course, you will be able to understand how websites and servers function, diverse data extraction techniques, and methods of handling and organizing data.

This Web Scraping course covers the following topics:

  • Review of data structures (Lists, Dictionaries, Tuples, File Handling)
  • How websites are hosted on servers
  • Calls to the server (GET, POST methods)
  • Review of HTML and CSS
  • Requests Module and BeautifulSoup Module overview
  • Parsing HTML using BeautifulSoup
  • Filtering elements using BeautifulSoup and navigating the Parse Tree
  • JavaScript and AJAX overview
  • Selenium and the need for it
  • Selecting elements using Selenium 
  • CSS selectors 
  • XPath selectors 
  • Navigating pages using Selenium 
  • Practical Projects

Who this course is for:
  • Those who want to learn how to use Python for web scraping and data extraction.