Modern Web Scraping with Python using Scrapy Splash Selenium
4.7 (1,342 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
7,817 students enrolled

Modern Web Scraping with Python using Scrapy Splash Selenium

Become an expert in web scraping and web crawling using Python 3, Scrapy, Splash and Selenium 2nd EDITION (2020)
Bestseller
4.7 (1,338 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
7,813 students enrolled
Created by Ahmed Rafik
Last updated 5/2020
English
English [Auto-generated], Italian [Auto-generated]
Current price: $16.99 Original price: $24.99 Discount: 32% off
5 hours left at this price!
30-Day Money-Back Guarantee
This course includes
  • 8.5 hours on-demand video
  • 35 articles
  • 8 downloadable resources
  • Full lifetime access
  • Access on mobile and TV
  • Assignments
  • Certificate of Completion
Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • Understand the fundamentals of Web Scraping
  • Scrape websites using Scrapy
  • Understand Xpath & CSS Selectors
  • Build a complete Spider from A to Z
  • Store the extracted Data in MongoDb & SQLite3
  • Scrape JavaScript websites using Splash & Selenium
  • Build a CrawlSpider
  • Understand the Crawling behavior
  • Build a custom Middleware
  • Web Scraping best practices
  • Avoid getting banned while scraping websites
  • Bypass cloudflare
  • Scrape APIs
  • Scrape infinite scroll websites
  • Working with Cookies
  • Deploy spiders locally and to the cloud
  • Run spiders periodically
  • Prevent storing duplicated data
  • Build datasets
  • Login to websites using Scrapy
  • Download images and files using Scrapy
Course content
Expand all 120 lectures 08:34:19
+ Introduction
5 lectures 17:04
Add VSCODE to path (Mac users)
00:24
Udemy 101 (Please don't skip*)
01:21
Asking questions
00:27
+ Scrapy Fundamentals
5 lectures 30:26
Scrapy fundamentals PART 1
05:09
Scrapy fundamentals PART 2
07:40
Scrapy fundamentals PART 3
06:35
Scrapy fundamentals PART 4
07:19
Scrapy fundamentals PART 5
03:43
+ XPath expressions & CSS Selectors
8 lectures 36:06

Hi everyone,

Due to some technical issues from Udemy side, some of you couldn't download the resources included in lecture 11 & 14. If that's the case for you please use the links I included below to download the resources needed for this section:

For lecture 11:

https://www.dropbox.com/s/bqrcr7a7vln0qwq/css_html_file.zip?dl=0

For lecture 14:

https://www.dropbox.com/s/zk2dbqr2b1talms/xpath_html_file.zip?dl=0

Please note, this article will be removed as soon as Udemy fixes this issue.

Kind regards,

Ahmed.

Downloadable files
00:15
XPath & CSS Selectors
02:53
CSS Selectors fundamentals
09:13
CSS selectors in theory
02:54
XPath fundamentals
08:47
Navigating using XPath(Going UP)
05:15
Navigating using XPath(Going DOWN)
03:23
XPath in theory
03:26
+ Project 1 Spiders from A to Z
6 lectures 21:18
Worldometers PART 1
04:26
Worldometers PART 2
05:16
Worldometers PART 3
06:53
Worldometers PART 4
03:57
Project source code
00:03
Exercise
00:43
+ Project 2 Dealing with Multiple pages
7 lectures 22:18
Website URL
00:14
Setting up the project
04:11
Building the spider
06:48
Dealing with pagination
03:41
Spoofing request headers
06:50
TinyDeal project source code
00:03
Exercise 2
00:31
+ Debugging spiders
2 lectures 13:21
Debugging spiders PART 1
09:09
Debugging spiders PART 2
04:12
+ Let's take a break !
2 lectures 04:29
The "whys" & "whens" of web scraping
02:50
Web scraping challenges
01:39
+ Project 3 Build Crawlers using Scrapy
6 lectures 20:37
Crawl spider structure
06:05
The Rule object
07:00
Following links in pagination
02:43
Spoofing request headers
04:13
Project source code
00:03
Exercise
00:33
+ Splash crash course
7 lectures 30:45
What dilemma splash came to solve
02:29
Setting up Splash (Windows Pro/Entreprise edition & Mac Os)
06:32
Setting up Splash(Windows Home Edition)
03:37
Setting up Splash (Linux)
01:24
Introduction to Splash
06:22
Working with elements
05:40
Spoofing request headers
04:41
Requirements
  • Basics of Python
  • Internet access
Description

Web Scraping nowadays has become one of the hottest topics, there are plenty of paid tools out there in the market that don't show you anything how things are done as you will be always limited to their functionalities as a consumer.

In this course you won't be a consumer anymore, i'll teach you how you can build your own scraping tool ( spider ) using Scrapy.

You will learn:

  1. The fundamentals of Web Scraping

  2. How to build a complete spider

  3. The fundamentals of XPath & CSS Selectors

  4. How to locate content/nodes from the DOM using XPath & CSS

  5. How to store the data in JSON, CSV... and even to an external database(MongoDb & SQLite3)

  6. How to write your own custom Pipeline

  7. Fundamentals of Splash

  8. How to scrape Javascript websites using Scrapy Splash & Selenium

  9. The Crawling behavior

  10. How to build a CrawlSpider

  11. How to avoid getting banned while scraping websites

  12. How to build a custom Middleware

  13. Web Scraping best practices

  14. How to scrape APIs

  15. How to use Request Cookies

  16. How to scrape infinite scroll websites

  17. Host spiders in Heroku for free

  18. Run spiders periodically with a custom script

  19. Prevent storing duplicated data

  20. Deploy Splash to Heroku

  21. Write data to Excel files

  22. Login to websites using Scrapy

  23. Download Files & Images using Scrapy

  24. Use Proxies with Scrapy Spider

  25. Use Crawlera with Scrapy & Splash

  26. Use Proxies with CrawlSpider


What makes this course different from the others, and why you should enroll ?

  • First, this is the most updated course. You will be using Python 3.7, Scrapy 1.6 and Splash 3.0

  • You will have an in-depth step by step guide on how to become a professional web scraper.


  • You will learn how to use Splash & Selenium to scrape JavaScript websites and I can assure you, you won't find any tutorials out there that teaches how to really use Splash like I'll be doing in this course.

  • You will learn how to host spiders in Heroku as well as Splash(Exclusive).

  • You will learn how to create a custom script so spiders can run periodically without any intervention from you.

  • 30 days money back guarantee by Udemy

So whether you are a data analyst who wants to add web scraping to his tool set or someone else who wants to learn how to extract unstructured data from unstructured HTML web pages and then store back that data in a structured way to apply some data analysis on it then you are welcome to join this course.

**STUDENTS THOUGHTS ABOUT THIS COURSE **

"I was particularly looking for web scraping using XPATHs and this course is addressing that. It also covers dynamic paging. A proper mix of theory and practical. A must-have for those who wants to do web scraping . GREAT learning experience !!! ". By Hiran Kumar

"90% of what I was searching for!!! Great job!! Clear explanations and great communication with Ahmed". By Raylyson Estanista 

"Admed’s Web scraping course is awesome . His approach using Python with scrapy and splash works well with all websites especially those that make heavy use of JavaScript. Ahmed is a gifted educator: expert communicator, passionate, conscientious and accessible to his students. I highly recommend this course and any of Ahmed Rafik’s Udemy courses. ". By Richard Blackmon

"Great course, and a nice introduction to Scrapy (I'm someone with no Python experience whatsoever).". By I S

"Excellent course. Quick and thorough at the same time. Ahmed is incredibly responsive to the students and often replies to questions within minutes! Highest recommendation." By Robert Nolte

"That course is very good and explanation is crystal clear! The instructor is very supportive in case of questions. Highly recommended." By Shubina Ekaterina

"I like the course. Clear explanations and good comunication with Ahmed. All topics is interesting and full of information. I improved my skils in Scrapy. Author update course content by new videos. It's a big bonus) Explained more advance topics I never see in other courses. Thank you, Ahmed. Waiting for new videos)". By Ruslan Romanenko



Who this course is for:
  • Anyone who wants to scrape data from any website
  • Anyone who wants to learn Scrapy
  • Anyone who wants to automate the task of copying contents from websites
  • Anyone who wants to learn how to scrape Javascript websites using Scrapy-Splash & Selenium