The Ultimate Web Scraping With Python Bootcamp 2024

Learn how HTTP status codes classify server responses for web scraping, from 1xx informational to 5xx errors, with examples like 200 OK, 301 moved, and 404 not found.

Explore how request headers identify clients, influence access controls, and how simulating a Google crawler demonstrates header-driven scraping concepts and the ethics of access.

Explore how proxies act as intermediaries between client and server, including caching proxies and load balancers, and how residential proxies rotate IP addresses to obfuscate scraping requests.

Master the basics of HTML markup, understand semantic versus structural tags, and learn how proper tagging improves accessibility and search engine understanding for effective web scraping.

Explore how JavaScript adds behavior to static web pages by linking a button to hide a link, using DOM selection, event listeners, and simple show-hide logic.

JavaScript drives dynamic HTML and CSS, enabling single page apps; learn to scrape such pages by using headless browsers to render and extract content.

Explore how to embed CSS and JavaScript directly in HTML using style and script tags, including inline styles and external scripting, to build a single, ready-to-scrape web page.

Learn how authentication verifies identity and authorization grants access, and implement common methods like API keys in headers, bearer tokens, and basic auth for API requests.

Explore beyond get: send delete, post, put, patch, head, and options requests using the requests library and HTTP bin, which echoes back your requests for testing.

Learn sending data in the request body with the Python requests library, comparing form encoded data and json payloads, and understanding their encoding and content type.

Navigate the parsed HTML tree with tags and attributes in beautiful soup, explore multi-valued attributes like class, and learn to select nested elements for data extraction.

Learn how html tags form a hierarchy from root to children and descendants, including ul, li, and a, using beautiful soup to navigate, filter navigable strings, and access relationships.

Learn how to navigate siblings in the Beautiful Soup navigation API, horizontal navigation to move between tags at the same tree depth using next_sibling and previous_sibling for efficient web scraping.

Extract all text content from HTML using Beautiful Soup's stripped strings and dot strings attributes, compare their results, and understand when to preserve or discard whitespace.

Refine the book data extraction by cleaning price strings to a float, using regex substitutions, and mapping ratings to integers, returning dictionaries with title, price, and rating.

Master pandas by turning a list of dictionaries into a dataframe for efficient data handling. Filter with boolean masks, calculate averages, and export books data to CSV, JSON, or Excel.

Target page elements by id or attribute using Beautiful Soup's find and find_all, then build complex filters with anonymous functions and lambda expressions for flexible scraping.

Explore find and find all, plus select and select one, to see how single-tag results differ from list results and how to verify element identity.

Learn how stocks, tickers, and exchanges map to a portfolio, fetch prices from Google Finance, manage multi-exchange pricing, and convert currencies for a unified view.

Use Python dataclasses to define stock, position, and portfolio types with ticker, exchange, currency, and price fields; employ post_init to fetch price data and prepare portfolio valuation.

Define position and portfolio as data classes to pair stocks with quantities and compute the portfolio's total value in USD from stock prices sourced via Google Finance.

Master the network tab to monitor HTTP requests, headers, and responses, then replicate server requests in Python to scrape data without parsing HTML or CSS.

Explore how to discover coffee shop locations by inspecting network requests, identifying GraphQL post queries that return restaurant data in JSON, and reproducing them with a Python workflow.

Replicate a web scraping request to source open positions from a career page using Python, intercept and replicate requests, and return position details with location, type, distance, and requirements.

Explore the adjacent sibling combinator, signified by the plus sign, to select elements that are siblings of a given element. Compare it with the general sibling combinator to choose wisely.

Master simple, compound, and complex selectors and selector lists, and use descendant, child, and sibling combinators to target elements for web scraping with precision.

Write a Python scraper that downloads the highest resolution Unsplash images by keyword, excluding premium watermarked ones, using both HTML and API approaches with pagination, saving images locally.

Prospect the site by analyzing image search results, filtering ads and premium images, and inspecting HTML structure to identify reliable selectors; compare HTML parsing with the API for high‑resolution images.

Explore calling the Unsplash API with Python, fetch images via get requests, parse photos results to extract full-resolution image URLs, and handle pagination for future downloads.

Learn how to paginate with api requests, constructing urls with page and per page, filtering by photos, and looping through pages to collect a target number of images.

Learn to render a javascript-heavy site with playwright and select legs, install chromium, and extract the budget value by waiting for network idle and parsing the html content.