Scrape the Planet! Building Web Scrapers with Python

Understand how browsers request pages and return html, forming the ui you see. Explore foundational scraper concepts, including page and api scrapers, plus ethics, legality, and design before coding.

Understand what a page scraper is, how it navigates HTML with tags, classes, and IDs, and how Python with Beautiful Soup extracts data.

Discover how APIs provide clean JSON data via endpoints and API keys, versus page scrapers, and learn when to use each.

Learn the ethics and legality of web scraping, including server load, rate limits, data licensing, and responsible use of scraped data to avoid harming sites and violating licenses.

Define a problem and scope to guide scraper design and manage the project's complexity. Plan data gathering via APIs or web scraping, and choose storage with MongoDB or post graphs.

Design a scraper using nouns like users and posts. Apply adjectives and relationships, define verbs, and skeleton code to plan data and storage for the next section.

Learn to use the Python requests library to fetch Yahoo Finance pages, perform get and post requests, and inspect response status codes.

Learn to fetch web pages with requests, parse HTML with BeautifulSoup, and extract stock prices and deltas from Yahoo Finance, with robust error handling.

Learn to scrape IMDb to build actor filmographies using requests and Beautiful Soup. Handle search results, navigate to actor pages, extract filmography data, and save results as JSON files.

Set up a local Postgres database to store scraped data by installing Postgres, creating a user and database, connecting with psql, and noting it's not secure for internet deployment.

Connect to a PostgreSQL database, create the stock_data table if needed, and store Yahoo-scraped stock data using psycopg2. Configure the database name, user, password, and host.

Review this recap as you celebrate your move from zero scraping knowledge to storing stock and IMDb data in databases. Get ready for faster spidering and the Kevin Bacon problem.

Explore the concepts of web spiders and how they differ from scrapers, focusing on data discovery, site-wide links, and depth-limited crawling using examples like IMDb and Reddit.

Introduce the Kevin Bacon problem through an IMDb spider, using depth-first and breadth-first search to connect actors via movies, and outline a Python web scraper with PostgreSQL storage.

Design a Kevin Bacon scraper in Python using a breadth-first search, with skeleton code, modular functions, and depth-limit caching.

Build the scraping muscle of the Kevin Bacon spider by implementing data retrieval, caching, and actor/movie data structures, with error handling and debugging tips for robust IMDb scraping.

Improve the IMDb spider by caching to reduce external requests and using a local backend cache to speed data retrieval and optimize route nodes.

Implement local caching in the IMDb spider to store actor data and movie casts, speeding up the Kevin Bacon spider by reducing external requests and leveraging a local database.

Build a simple Wikipedia spider from scratch in Python, using requests and Beautiful Soup to crawl internal links and extract the page title and links for storage in Postgres.

Recap of section four on spiders and the Kevin Bacon problem, with a Wikipedia spider, then preview section five's smart queues and multi-threading for scalable web scrapers.

Design a one-stock price watcher that scrapes a stock page, checks every 30 seconds, logs prices to a database, and triggers alerts when price hits 48 or 52.

Design a stock price watcher in Python that scrapes Yahoo Finance, stores prices with timestamps in a database, and alerts when the price crosses set high or low thresholds.

Accelerate your stock price watcher in Python by using multithreading to process a watch list with batched stock data, while managing concurrency and avoiding race conditions.

Master the theory of enterprise-scale spiders and scrapers using a database-backed job queue, where spiders discover content and scrapers process posts, users, and comments.

Explore prioritized job queues that coordinate spiders and scrapers, with a queue of diverse jobs. Apply dynamic priority scoring based on user activity and lexical analysis to prioritize processing.

Concludes section five by tying together multithreaded stock price watchers and queue data structures, and previews section six's call to action for building more powerful web scrapers.

Finish this Python web scraping course and apply your spiders and scrapers to real-world projects, from IMDb crawls to stock watchers. Use your skills for good to advance research.