Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Software Development Tools No-Code Development
Business
Entrepreneurship Communication Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certifications Network & Security Hardware Operating Systems & Servers Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Paid Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement & Gardening Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition & Diet Yoga Mental Health Martial Arts & Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Learning Teacher Training Test Prep Other Teaching & Academics
Web Development JavaScript React Angular CSS Node.Js PHP HTML5 Vue JS
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Amazon AWS Cisco CCNA Microsoft AZ-900 AWS Certified Developer - Associate
Microsoft Power BI SQL Tableau Data Modeling Business Analysis Business Intelligence MySQL Qlik Sense Blockchain
Unity Unreal Engine Game Development Fundamentals C# 3D Game Development C++ Unreal Engine Blueprints 2D Game Development Virtual Reality
Google Flutter Android Development iOS Development React Native Swift Dart (programming language) Mobile App Development Kotlin SwiftUI
Graphic Design Photoshop Adobe Illustrator Drawing Digital Painting Canva InDesign Character Design Procreate Digital Illustration App
Life Coach Training Neuro-Linguistic Programming Personal Development Personal Transformation Life Purpose Mindfulness Meditation CBT Cognitive Behavioral Therapy Sound Therapy
Entrepreneurship Fundamentals Business Fundamentals Freelancing Business Strategy Startup Business Plan Online Business Blogging Home Business
Digital Marketing Social Media Marketing Marketing Strategy Internet Marketing Google Analytics Copywriting Email Marketing YouTube Marketing Drop Servicing

IT & SoftwareOther IT & SoftwareWeb Scraping

Web Scraping 101 with Python3 using REQUESTS, LXML & SPLASH

Web Scraping for absolute beginners with Python using Requests, the powerful LXML and Splash for JavaScript websites
Rating: 4.3 out of 54.3 (357 ratings)
2,930 students
Created by Ahmed Rafik
Last updated 3/2020
English
English [Auto]

What you'll learn

  • LXML core fundamentals
  • XPath & CSS selectors
  • How send HTTP requests with Python
  • Scraping HTML web pages
  • Scraping multiple pages using recursion
  • Scraping APIs
  • Splash HTTP API
  • Scraping JavaScript websites using Splash
  • Authentication and Login to websites using Requests
  • Web scraping best practices
  • Building datasets

Requirements

  • Basic level of Python
  • PC with Internet connection

Description

What is web scraping ?

Let's say your boss at work gave you a task where he wants you to extract about 1000 product from a website, structure the data and save it to a database, would you copy paste manually all the product details from the product name, url and price ? I can imagine you would work days and nights and you wont finish the task, so this is where web scraping shines. So web scraping, or web harvesting or web data extraction is like writing a script that will automate data extraction from websites in a matter of minutes !.

Why learn Web Scraping ?

Whether you're a data analyst, a web developer or even someone who wants to work as a freelancer you should learn web scraping. 

For a data analyst building a dataset is extremely important, so without web scraping you simply can't generate it in addition to that adding web scraping in your resume is a plus for you.

Web scraping can be used in a variety of fields, so let me give you some examples on what you can do with it:

  1. Generate leads,

  2. Drop shipping where basically you gonna constantly scrape products from different online stores and show case them on your website to make money,

  3. Monitor products prices to get the best deals,

  4. automation,

  5. Machine learning,

  6. Web scraping freelancer

Of course there are tons and tons and variety of fields where web scraping can be extremely beneficial.

Is this course the right one for you ?

I've carefully planned and designed this course to be beginner friendly, from my experience I know those who do web scraping are mostly data analyst with no background knowledge on how the web works, how requests are made, how to locate and parse the data from the web and much much more, in addition to that this the most updated course regarding the material included and the tools used, so in this course

  1. I'll introduce to you the most used web scraping tools/frameworks

  2. We will setup the development environment from scratch

  3. You will learn and understand LXML core fundamentals

  4. How to use XPath & CSS selectors to select the data from a web page

  5. How the web works (Request/Response)

  6. How to scrape simple HTML web pages

  7. How to scrape multiple web pages

  8. Extract data from APIs

  9. You will learn Splash(crash course) so you can use it to scrape JavaScript websites

  10. Authentication/Login

  11. Store the extracted data whether to JSON/CSV files or MongoDb/SQLite3

  12. Exclusive tips and tricks regarding web scraping

Finally this course is project based, each section starting from the 2nd one we will experiment with a different website, each project has a certain degree of difficulty  and each one is completely independent from other projects. 

Is there is any assignments/exercises included in this course ? 

Yes, each section has an assignment included to it, this will help to get your hands dirty and by the end of each section after doing the assignment included you will feel more confident and comfortable with web scraping.

Why LXML and not BeautifulSoup ?

LXML is a lightweight HTML parser even the most popular web scraping framework (Scrapy) is built on the top of LXML, BeautifulSoup is a little bit overloaded with the number of functions exposed to us, it has more functions to use, yes that's right ! however in Web Scraping most of the time we use XPath and CSS Selectors to navigate and select what to scrape from the HTML web page (tree) so there is no need to learn about new functions and wasting all that time to familiarize yourself with the BeautifulSoup API and the internal architecture, in addition to all of that LXML in terms of performance is way better than BeautifulSoup.

Who is your instructor ? 

Hi! I'm Ahmed nice to meet you, my students prefer to call me web scraping Ninja and currently I have taught more than 2000 students around the world how to do web scraping. I personally do web scraping on daily basis whether for fun, for personal projects or as a freelancer and guess what ? I even have a master degree in computer science.

Should I enroll to this course ?

Honestly, by enrolling to this course you have nothing to lose, because if this course didn't meet your requirements, you can always ask for a refund in less than 30 days from the day you enrolled to the course guaranteed by Udemy with NO QUESTION TO ASK !

SO IF YOU DON'T KNOW ANYTHING ABOUT WEB SCRAPING & YOU DON'T KNOW WHERE TO START  ENROLL NOW ! :)

Who this course is for:

  • Anyone wants to learn Web scraping using Python, Requests and LXML
  • Anyone wants to learn how to use Splash to scrape JavaScript websites
  • Complete beginners with no background on web scraping
  • Those who already have basic familiarity with web scraping and want to fill the gaps

Featured review

Dan E
Dan E.
106 courses
8 reviews
Rating: 5.0 out of 5a year ago
Have been a developer for 10 years. This is probably the only realistic way, and also the best way, that I've seen to learn web scraping on the whole internet. Web scraping began going through a huge renaissance last year, if not a little bit earlier. It's really important to know now and can allow you to do so many cool things. Please buy lol

Instructor

Ahmed Rafik
Developer and Online Instructor
Ahmed Rafik
  • 4.4 Instructor Rating
  • 3,946 Reviews
  • 28,228 Students
  • 3 Courses

Who I am?

I’m Ahmed Rafik, I'm a self-taught developer & an online teacher on Udemy. I've helped thousands of people learning web scraping with Python using different tools such as Scrapy, Splash & Selenium.

As a self-taught developer, I found myself jumping between different tutorials and (e-)books trying to understand how things can work out together, I was literally wasting so much time trying to connect the dots rather than learning how to code and over time this has become so overwhelming with lots of things to learn and to connect.

I believe coding should be easy for everyone but this also requires you to choose the right instructor with the right knowledge.

In my courses, I'm gonna teach you the skills you need to start your web scraping career from the get-go with no fluff. I try as much as possible to avoid the boring theoretical explanations unless it's necessary, I always keep my courses up to date and that's what helped me to have the highest-rated and the best selling web scraping courses on Udemy.

I can't wait to see you enrolled in one of my courses, I'll make sure to be there for you on every step you make and answer any questions you have.

Top companies choose Udemy Business to build in-demand career skills.
NasdaqVolkswagenBoxNetAppEventbrite
  • Udemy Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Investors
  • Impressum Kontakt
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Accessibility statement
Udemy
© 2022 Udemy, Inc.