Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Development Tools No-Code Development
Business
Entrepreneurship Communications Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certification Network & Security Hardware Operating Systems Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design Design Thinking 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition Yoga Mental Health Dieting Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Teacher Training Test Prep Other Teaching & Academics
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Cisco CCNA Amazon AWS CompTIA Security+ Microsoft AZ-900
Graphic Design Photoshop Adobe Illustrator Drawing Digital Painting InDesign Character Design Canva Figure Drawing
Life Coach Training Neuro-Linguistic Programming Personal Development Mindfulness Personal Transformation Life Purpose Meditation CBT Emotional Intelligence
Web Development JavaScript React CSS Angular PHP Node.Js WordPress Vue JS
Google Flutter Android Development iOS Development React Native Swift Dart Programming Language Mobile Development Kotlin SwiftUI
Digital Marketing Google Ads (Adwords) Social Media Marketing Google Ads (AdWords) Certification Marketing Strategy Internet Marketing YouTube Marketing Email Marketing Retargeting
Microsoft Power BI SQL Tableau Business Analysis Data Modeling Business Intelligence MySQL Data Analysis Blockchain
Business Fundamentals Entrepreneurship Fundamentals Business Strategy Business Plan Startup Online Business Freelancing Blogging Home Business
Unity Game Development Fundamentals Unreal Engine C# 3D Game Development C++ 2D Game Development Unreal Engine Blueprints Blender
30-Day Money-Back Guarantee
Development Programming Languages Web Scraping

Modern Web Scraping with Python using Scrapy Splash Selenium

Become an expert in web scraping and web crawling using Python 3, Scrapy, Splash and Selenium 2nd EDITION (2020)
Bestseller
Rating: 4.7 out of 54.7 (2,401 ratings)
13,823 students
Created by Ahmed Rafik
Last updated 1/2021
English
English [Auto], Italian [Auto]
30-Day Money-Back Guarantee

What you'll learn

  • Understand the fundamentals of Web Scraping
  • Scrape websites using Scrapy
  • Understand Xpath & CSS Selectors
  • Build a complete Spider from A to Z
  • Store the extracted Data in MongoDb & SQLite3
  • Scrape JavaScript websites using Splash & Selenium
  • Build a CrawlSpider
  • Understand the Crawling behavior
  • Build a custom Middleware
  • Web Scraping best practices
  • Avoid getting banned while scraping websites
  • Bypass cloudflare
  • Scrape APIs
  • Scrape infinite scroll websites
  • Working with Cookies
  • Deploy spiders locally and to the cloud
  • Run spiders periodically
  • Prevent storing duplicated data
  • Build datasets
  • Login to websites using Scrapy
  • Download images and files using Scrapy
Curated for the Udemy for Business collection

Course content

18 sections • 127 lectures • 8h 50m total length

  • Preview06:47
  • Setting up Scrapy the Development Environment (Updated)
    Preview08:05
  • Add VSCODE to path (Mac users)
    00:26
  • Udemy 101 (Please don't skip*)
    01:21
  • Asking questions
    00:27

  • Scrapy fundamentals PART 1
    05:09
  • Scrapy fundamentals PART 2
    07:40
  • Scrapy fundamentals PART 3
    06:35
  • Scrapy fundamentals PART 4
    07:19
  • Scrapy fundamentals PART 5
    03:43

  • Downloadable files
    00:15
  • XPath & CSS Selectors
    02:53
  • CSS Selectors fundamentals
    09:13
  • CSS selectors in theory
    02:54
  • XPath fundamentals
    08:47
  • Navigating using XPath(Going UP)
    05:15
  • Navigating using XPath(Going DOWN)
    03:23
  • XPath in theory
    03:26

  • Worldometers PART 1
    04:26
  • Worldometers PART 2
    05:16
  • Worldometers PART 3
    06:53
  • Worldometers PART 4
    03:57
  • Project source code
    00:03
  • Exercise
    00:43

  • Bulding datesets
    04:23

  • Website URL (Please do not skip)
    00:50
  • Setting up the project
    04:11
  • Setting up the project - Code update -
    00:12
  • Building the spider
    06:48
  • Dealing with pagination
    03:41
  • Spoofing request headers
    06:50
  • TinyDeal project source code
    00:03
  • Exercise 2
    00:31

  • What is debugging?
    01:48
  • Debugging spiders PART 1
    09:09
  • Debugging spiders PART 2
    04:12

  • The "whys" & "whens" of web scraping
    02:50
  • Web scraping challenges
    01:39

  • Crawl spider structure
    06:05
  • The Rule object
    07:00
  • Following links in pagination
    02:43
  • Spoofing request headers
    04:35
  • Project source code
    00:03
  • Exercise
    00:33

  • What dilemma splash came to solve
    02:29
  • Setting up Splash (Windows Pro/Entreprise edition & Mac Os)
    06:32
  • Setting up Splash(Windows Home Edition)
    03:37
  • Setting up Splash (Linux)
    01:24
  • Introduction to Splash
    06:22
  • Working with elements
    05:40
  • Spoofing request headers
    04:41

Requirements

  • Basics of Python
  • Internet access

Description

Web Scraping nowadays has become one of the hottest topics, there are plenty of paid tools out there in the market that don't show you anything how things are done as you will be always limited to their functionalities as a consumer.

In this course you won't be a consumer anymore, i'll teach you how you can build your own scraping tool ( spider ) using Scrapy.

You will learn:

  1. The fundamentals of Web Scraping

  2. How to build a complete spider

  3. The fundamentals of XPath & CSS Selectors

  4. How to locate content/nodes from the DOM using XPath & CSS

  5. How to store the data in JSON, CSV... and even to an external database(MongoDb & SQLite3)

  6. How to write your own custom Pipeline

  7. Fundamentals of Splash

  8. How to scrape Javascript websites using Scrapy Splash & Selenium

  9. The Crawling behavior

  10. How to build a CrawlSpider

  11. How to avoid getting banned while scraping websites

  12. How to build a custom Middleware

  13. Web Scraping best practices

  14. How to scrape APIs

  15. How to use Request Cookies

  16. How to scrape infinite scroll websites

  17. Host spiders in Heroku for free

  18. Run spiders periodically with a custom script

  19. Prevent storing duplicated data

  20. Deploy Splash to Heroku

  21. Write data to Excel files

  22. Login to websites using Scrapy

  23. Download Files & Images using Scrapy

  24. Use Proxies with Scrapy Spider

  25. Use Crawlera with Scrapy & Splash

  26. Use Proxies with CrawlSpider


What makes this course different from the others, and why you should enroll ?

  • First, this is the most updated course. You will be using Python 3.7, Scrapy 1.6 and Splash 3.0

  • You will have an in-depth step by step guide on how to become a professional web scraper.


  • You will learn how to use Splash & Selenium to scrape JavaScript websites and I can assure you, you won't find any tutorials out there that teaches how to really use Splash like I'll be doing in this course.

  • You will learn how to host spiders in Heroku as well as Splash(Exclusive).

  • You will learn how to create a custom script so spiders can run periodically without any intervention from you.

  • 30 days money back guarantee by Udemy

So whether you are a data analyst who wants to add web scraping to his tool set or someone else who wants to learn how to extract unstructured data from unstructured HTML web pages and then store back that data in a structured way to apply some data analysis on it then you are welcome to join this course.

**STUDENTS THOUGHTS ABOUT THIS COURSE **

"I was particularly looking for web scraping using XPATHs and this course is addressing that. It also covers dynamic paging. A proper mix of theory and practical. A must-have for those who wants to do web scraping . GREAT learning experience !!! ". By Hiran Kumar

"90% of what I was searching for!!! Great job!! Clear explanations and great communication with Ahmed". By Raylyson Estanista 

"Admed’s Web scraping course is awesome . His approach using Python with scrapy and splash works well with all websites especially those that make heavy use of JavaScript. Ahmed is a gifted educator: expert communicator, passionate, conscientious and accessible to his students. I highly recommend this course and any of Ahmed Rafik’s Udemy courses. ". By Richard Blackmon

"Great course, and a nice introduction to Scrapy (I'm someone with no Python experience whatsoever).". By I S

"Excellent course. Quick and thorough at the same time. Ahmed is incredibly responsive to the students and often replies to questions within minutes! Highest recommendation." By Robert Nolte

"That course is very good and explanation is crystal clear! The instructor is very supportive in case of questions. Highly recommended." By Shubina Ekaterina

"I like the course. Clear explanations and good comunication with Ahmed. All topics is interesting and full of information. I improved my skils in Scrapy. Author update course content by new videos. It's a big bonus) Explained more advance topics I never see in other courses. Thank you, Ahmed. Waiting for new videos)". By Ruslan Romanenko



Who this course is for:

  • Anyone who wants to scrape data from any website
  • Anyone who wants to learn Scrapy
  • Anyone who wants to automate the task of copying contents from websites
  • Anyone who wants to learn how to scrape Javascript websites using Scrapy-Splash & Selenium

Featured review

Visitor Code
Visitor Code
136 courses
18 reviews
Rating: 4.5 out of 510 months ago
Ahmed is super knowledgeable and helpful. The course is full of great information and you'd be glad to have taken it. I'm on both this and the advanced level courses. Even though at times it may seem a little complicated and the pieces aren't clearly connected, Ahmed is actively responding to all questions, unlike many course-abandoning instructors.

Instructor

Ahmed Rafik
Developer and Online Instructor
Ahmed Rafik
  • 4.7 Instructor Rating
  • 2,909 Reviews
  • 21,190 Students
  • 3 Courses

Who I am?

I’m Ahmed Rafik, I'm a self-taught developer & an online teacher on Udemy. I've helped thousands of people learning web scraping with Python using different tools such as Scrapy, Splash & Selenium.

As a self-taught developer, I found myself jumping between different tutorials and (e-)books trying to understand how things can work out together, I was literally wasting so much time trying to connect the dots rather than learning how to code and over time this has become so overwhelming with lots of things to learn and to connect.

I believe coding should be easy for everyone but this also requires you to choose the right instructor with the right knowledge.

In my courses, I'm gonna teach you the skills you need to start your web scraping career from the get-go with no fluff. I try as much as possible to avoid the boring theoretical explanations unless it's necessary, I always keep my courses up to date and that's what helped me to have the highest-rated and the best selling web scraping courses on Udemy.

I can't wait to see you enrolled in one of my courses, I'll make sure to be there for you on every step you make and answer any questions you have.

  • Udemy for Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Impressum Kontakt
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Featured courses
Udemy
© 2021 Udemy, Inc.