Web Scraping in Python Requests, Scrapy, Selenium, AI - 2026

Get started with Python’s most popular scraping tool — Requests. Learn how it works, why it's ideal for simple scraping, and how it compares to heavier tools like Scrapy. A clean intro to real-world HTTP handling.

Dive into the structure of web pages and learn how to precisely grab data using XPath with the lxml library. We'll extract real product links and teach you how DOM navigation works.

Learn to scrape car listings using Requests and XPath. Extract links, handle pagination, and save car details into a CSV — step-by-step.

Websites can block bots — unless you know how to disguise your scraper. Learn how to use headers to mimic real browsers and reduce your chances of getting flagged.

Some of the cleanest data lives in JSON on some websites. Learn to parse that JSON directly using Requests — skipping messy HTML and Xpaths altogether.

Websites don’t make scraping easy — they use rate limiting, CAPTCHAs, JavaScript challenges, TLS fingerprinting, honeypots, and more. In this lesson, you’ll learn how modern scrapers overcome these defenses using tools like proxies, headless browsers, stealth headers, and CAPTCHA solvers. We also clarify the legal and ethical boundaries of scraping, including a breakdown of the HiQ vs. LinkedIn case.

Learn why scraping HTML isn’t enough. This lesson introduces network activity — the hidden stream of API calls, headers, and cookies your browser makes on every page load. You’ll discover how inspecting network traffic reveals clean, structured data like prices, stock levels, and product info — often easier to scrape than messy HTML. This is the foundation of pro-level scraping.

In this practical lesson, you’ll learn how to extract structured data from a real public API — without touching the front-end. We’ll reverse-engineer the “Find a Doctor” page on the USA Health website using DevTools, identify the JSON endpoint, inspect headers, decode the POST payload, and write a full Python scraper using requests. You'll also paginate through all results and save the data into a clean CSV — production-style.

Learn how to find hidden API data using Selenium Wire when DevTools fails. We'll extract car listing data from DTC Lease by capturing JS-triggered requests and scraping the revealed JSON endpoint.

Use MITMProxy to capture real browser traffic, uncover secret APIs, and extract headers or cookies for ultra-advanced scraping. It’s like DevTools on steroids.

Use Bright Data’s Web Unlocker to breeze past CAPTCHAs and login walls. A plug-and-play solution for when you just want to get the data — fast.

Sitemaps are goldmines. This lesson shows you how to find them, parse XML, and extract hundreds of product links with minimal effort — all with Requests.

Discover Scrapy — the Python framework built for large-scale web scraping. Learn how it compares to Requests, understand its modular structure, and kick off your first project using scrapy startproject, scrapy shell, and more.

Create your first Scrapy spider to scrape quotes and authors across multiple pages. Learn XPath selection, response.follow() for pagination, and the basics of running spiders and saving output in structured formats.

Learn to organize and store scraped data efficiently. Define custom Items, clean data in pipelines, and save to JSON or PostgreSQL — all without writing extra code each time. A real upgrade from manual scripts.

Deploy your spiders with Scrapyd or Scrapy Cloud. Learn to set up a local server, trigger spiders with curl, automate jobs with cron, and monitor runs on a dashboard — all designed for production scraping.

In this lesson, we improve our Wellfound scraper using Scrapy middleware to rotate user-agents and send requests through Bright Data proxies. You'll learn how middleware helps avoid IP blocks and adds pagination for multi-page scraping.

Use Bright Data proxy API
Rotate user-agents with scrapy-user-agents
Add pagination logic
Track logs to confirm bypass success

This builds on our earlier scraper with stronger anti-scraping tactics.

Learn what makes Selenium different from Requests and Scrapy. See how it automates real browsers like Chrome, making it ideal for dynamic, JavaScript-heavy sites. Get a feel for browser drivers, the WebDriver protocol, and when Selenium is your best option.

Build your first Selenium bot by automating a Google search. Type, click, extract titles, and close the browser — all with Python. Learn core concepts like element selection, navigation, and scraping with XPath.

Tackle a real dynamic site with dropdowns and JavaScript rendering. You'll build a resilient bot that loops through categories, scrapes processing times, and handles exceptions like a pro.

Tackle a real dynamic site with dropdowns and JavaScript rendering. You'll build a resilient bot that loops through categories, scrapes processing times, and handles exceptions like a pro.

Skip login pages with pre-saved cookies and scrape private Facebook group leads using regex. Learn three login strategies — cookies, browser profiles, and manual — and when to use each.

Bypass reCAPTCHA v2 using the 2Captcha API. Learn how to grab the sitekey, send requests, inject the token, and log in — all fully automated. Includes pricing tips and troubleshooting advice.

Add serious power to your scraper with IP rotation using Selenium Wire and Oxylabs proxies. Scrape foreclosure cases reliably using rotating IPs, error handling, and resume-friendly logic.

Avoid detection by rotating browser fingerprints with GoLogin or Undetectable.io. Learn how websites track you and how to spoof key identifiers to appear like unique, real users — legally and safely.

Use ChatGPT/Claude to draft scraper skeletons, refine selectors, and fix bugs fast. Learn prompt patterns that work and where AI won’t help (anti-bot defenses). Cut costs with free tiers, smaller models, caching, and local models via Ollama. We’ll also cover group-subscription sites like GoSplit—pros/cons, ToS risks—and safer alternatives. Leave with a lean, affordable AI workflow.

Cursor/Windsurf shine on large, multi-file scraper projects (repo-wide edits, refactors). For single-file jobs, they’re often slower than a simple stack (VS Code + ChatGPT/Claude + Repo Prompt). See quick demos and a decision checklist so you’ll know when an AI IDE saves hours—and when a lightweight flow is faster and cheaper.

Students will discover how Repo Prompt helps them understand any GitHub scraper repository instantly using AI, without reading code line-by-line. We’ll learn how to load a repo, ask questions about scraper logic, locate key files, and even generate context-aware prompts that save hours of manual digging.

This lecture breaks down Large Language Models (LLMs) from scratch in simple language – how they’re trained, what “large” means, how prompts really work, and why models like ChatGPT, Claude, and Gemini can write scrapers, parse HTML, and help us automate workflows. No maths — just practical insight for scrapers!

A practical comparison of paid/proprietary LLMs — OpenAI GPT-4, Claude 3.5, Google Gemini, Grok — focusing on their pricing, token limits, API setup, and which ones perform best for scraping tasks like code generation, HTML parsing and summarisation. Students learn how to select the right API for commercial use.

This lesson introduces the world of free / offline LLMs, teaching students how to run models locally with Ollama, download models like Mistral / LLaMA 3, and explore Hugging Face for fine-tuned public models. We’ll run a model entirely on-device — with zero cost — and prepare it for scraping workflows.

A complete hands-on project: scrape the first 100 reviews from IMDb using Selenium, and then pipe each review into Mistral (via Ollama) to generate Positive/Neutral/Negative sentiment. We’ll save results to CSV/JSON and discuss real-life applications: dashboards, client reports, trend analysis & more.

Turn combined_with_sellers.csv into hyper-personalized emails using Mistral via Ollama. For each seller, the prompt classifies performance (BSR trend, buyers, sold change, BSR level), selects an angle, writes a 110–140 word email with RootCategory + Title hook, PPC offer, variation note, Loom CTA, and 3 subject lines—returned as strict JSON. Save to CSV/JSONL; run a 5-row test, then scale.

Learn how to combine screenshots and vision-enabled LLMs to extract structured data from visual charts and pages. In this lesson, we walk through the workflow of capturing images, sending them to a model, and parsing the AI’s JSON output for practical scraping use cases.

Learn how to convert your Python scrapers into standalone executables that clients can run easily without setting up Python or libraries.

Step-by-step guide to hosting a scraper on Render’s platform, making your code accessible online with minimal server management.

Extend your scraper by uploading results directly into Google Sheets via Google Cloud Console, turning scrapes into live, shareable data.

Learn how to deploy Scrapy spiders to Zyte Scrapy Cloud, automate scheduling, and manage crawls in a production-ready environment.

Introduction to Docker—understand what it is, why it’s used, and how to install it on different systems to prepare for containerized scraping.

Break down the Docker boilerplate files (Dockerfile, .env, requirements.txt) and understand their role in building reliable scraper images.

Watch how to generate a Docker image from our scraper, run it locally, and share it online so it can be pulled and executed anywhere.