The Ultimate Web Scraping With Python Bootcamp 2024

Explore how HTTP serves as the foundation for data communication on World Wide Web. Describe its client and server request and response model and extensibility through headers and status codes.

Explore the HTTP request–response cycle using a tool that sends HTTP requests and reveals responses, highlighting how a client and server exchange stateless messages that render HTML in the browser.

Understand how http remains stateless yet the web feels stateful through cookies and server-stored data. See how login and cookies enable ongoing authentication across requests.

Explore HTTP verbs and their roles, including get, post, put, patch, and delete; see how get retrieves resources, post creates, put and patch update, and delete removes.

Learn how HTTP status codes classify server responses for web scraping, from 1xx informational to 5xx errors, with examples like 200 OK, 301 moved, and 404 not found.

Learn how http headers extend requests and responses, enabling cookies for session management and accept-language headers to render language-specific pages.

Explore how request headers identify clients, influence access controls, and how simulating a Google crawler demonstrates header-driven scraping concepts and the ethics of access.

Explore how proxies act as intermediaries between client and server, including caching proxies and load balancers, and how residential proxies rotate IP addresses to obfuscate scraping requests.

Learn how HTML, CSS, and JavaScript compose web pages and how browsers fetch and render them, using dev tools to inspect and modify elements for web scraping practice.

Master the basics of HTML markup, understand semantic versus structural tags, and learn how proper tagging improves accessibility and search engine understanding for effective web scraping.

Explore how html attributes extend tags to alter behavior and presentation, using ref for links and target to open in a new tab, and how id and class group elements.

Explore CSS basics for web scraping: select HTML elements with tag, class, and id selectors; style with color and text transform; hide elements with display none.

Learn how HTML class attributes group elements and how CSS selectors use dot notation to apply borders, colors, and decorations; explore multiple classes and the separation of markup from presentation.

Explore how JavaScript adds behavior to static web pages by linking a button to hide a link, using DOM selection, event listeners, and simple show-hide logic.

Learn to turn a click into a dynamic toggle that shows or hides a link with JavaScript, using a toggle link function, style.display, and updating the button text with textContent.

JavaScript drives dynamic HTML and CSS, enabling single page apps; learn to scrape such pages by using headless browsers to render and extract content.

Explore how HTML, CSS, and JavaScript compose web pages, with JavaScript as a general purpose language, and learn their distinct comment syntax for HTML, CSS, and JavaScript.

Explore how to embed CSS and JavaScript directly in HTML using style and script tags, including inline styles and external scripting, to build a single, ready-to-scrape web page.

Learn how to fetch web pages in Python using the urllib library, handle HTTP responses, decode byte content, and compare its simplicity to modern alternatives like requests for web scraping.

Install and import the requests library in Python to perform http get requests. Access response content with text or content, check status code 200, and read headers for scraping workflows.

Explore how to use the response and request objects to read and set http headers, including content type and case-insensitive keys, with the headers parameter.

Learn to build query parameters with the requests library using a base URL and a params dictionary, encoding values and improving readability; see Coinbase API and Sunrise API examples.

Learn how authentication verifies identity and authorization grants access, and implement common methods like API keys in headers, bearer tokens, and basic auth for API requests.

Explore beyond get: send delete, post, put, patch, head, and options requests using the requests library and HTTP bin, which echoes back your requests for testing.

Learn sending data in the request body with the Python requests library, comparing form encoded data and json payloads, and understanding their encoding and content type.

Discover how Beautiful Soup simplifies extracting data from HTML and XML by building a navigable soup object, enabling straightforward parsing and tag selection with Python.

Navigate the parsed HTML tree with tags and attributes in beautiful soup, explore multi-valued attributes like class, and learn to select nested elements for data extraction.

Learn how html tags form a hierarchy from root to children and descendants, including ul, li, and a, using beautiful soup to navigate, filter navigable strings, and access relationships.

Learn how to navigate siblings in the Beautiful Soup navigation API, horizontal navigation to move between tags at the same tree depth using next_sibling and previous_sibling for efficient web scraping.

Learn how to extract text from web-scraped tags using get_text, text, and string in BeautifulSoup, and understand when each returns content, including handling whitespace and separators.

Extract all text content from HTML using Beautiful Soup's stripped strings and dot strings attributes, compare their results, and understand when to preserve or discard whitespace.

Master web scraping with Beautiful Soup by using find and find_all to target tags, classes, and attributes, including price elements and data loading text with lambda-based filters.

Complete a challenge that fetches books page, builds a soup object, and parses titles, prices as floats, and ratings as integers into a list of dictionaries for first page.

Import requests and beautiful soup, fetch the page, locate each product pod, and extract title, price, and rating with targeted selectors and a helper function.

Refine the book data extraction by cleaning price strings to a float, using regex substitutions, and mapping ratings to integers, returning dictionaries with title, price, and rating.

Master pandas by turning a list of dictionaries into a dataframe for efficient data handling. Filter with boolean masks, calculate averages, and export books data to CSV, JSON, or Excel.

Target page elements by id or attribute using Beautiful Soup's find and find_all, then build complex filters with anonymous functions and lambda expressions for flexible scraping.

Master text-based searching in BeautifulSoup using the text argument, regex patterns, and case-insensitive matching; compare find_all with stripped_strings and target anchors for flexible data extraction.

Master CSS selectors with the Beautiful Soup select method to search the tree using single-line criteria like article.product > h3 > a and the child combinator.

Explore find and find all, plus select and select one, to see how single-tag results differ from list results and how to verify element identity.

Scrape live price data from Google Finance to evaluate a portfolio. Convert prices from different currencies to USD and compute market value and percent allocation.

Learn how stocks, tickers, and exchanges map to a portfolio, fetch prices from Google Finance, manage multi-exchange pricing, and convert currencies for a unified view.

Set up a Python scraping project with a virtual environment, install requests and BeautifulSoup, and build price scraper using data-last-price and data-currency attributes to return ticker, exchange, price, and currency.

Convert cross-currency stock prices to usd by fetching fx rates and applying them to cad or other currencies. Build an fx-to-usd helper and test with tsx and nyse prices.

Use Python dataclasses to define stock, position, and portfolio types with ticker, exchange, currency, and price fields; employ post_init to fetch price data and prepare portfolio valuation.

Define position and portfolio as data classes to pair stocks with quantities and compute the portfolio's total value in USD from stock prices sourced via Google Finance.

Display a stock portfolio in a polished tabular view using tabulate, showing ticker, exchange, quantity, price, market value, and allocation, with sorting and formatting.

Master the network tab to monitor HTTP requests, headers, and responses, then replicate server requests in Python to scrape data without parsing HTML or CSS.

Explore how to discover coffee shop locations by inspecting network requests, identifying GraphQL post queries that return restaurant data in JSON, and reproducing them with a Python workflow.

Learn why APIs speed up web scraping, reduce volatility, and offer customizable data by querying servers directly, avoiding heavy HTML parsing and relying on GraphQL when needed.

Emulate headers and replicate requests by exporting a browser request as curl and importing it into an HTTP client to send a post with the body and GraphQL parameters.

Compare Postman to httpie as an http client, import curl requests, save and organize headers and bodies in a collection, and export responses, with a plan to port to Python.

Export and port intercepted HTTP requests from the network tab to Python code using requests or HTTP client, via Postman and GraphQL queries, including handling headers and multi-line payloads.

Replicate a web scraping request to source open positions from a career page using Python, intercept and replicate requests, and return position details with location, type, distance, and requirements.

Learn to interact with an API by inspecting network requests, crafting curl and Postman calls, and building a Python function that fetches jobs by latitude and longitude and returns JSON.

Parse the API response data to extract job attributes such as title, full time, part time, requirements, and distance, then build a pandas dataframe and export to open positions.csv.

Explore geocoding in Python by converting postal codes to latitude and longitude with the pgeocode package, enabling offline location queries and integration with dataframe workflows.

Explore select legs, a parsing library for HTML and XML that serves as a high‑performance alternative to Beautiful Soup, emphasizing CSS selectors and improved speed from C libraries.

Install Selectolax and pull up the Wikipedia page HTML with requests, then parse it with the HTML parser and extract paragraphs via the tree.css CSS selector.

Explore CSS selectors and combinators, including type, class, ID, and attribute selectors, and master descendant and child combinators to select elements precisely, such as edit links inside headings, improving robustness.

Explore the adjacent sibling combinator, signified by the plus sign, to select elements that are siblings of a given element. Compare it with the general sibling combinator to choose wisely.

Master simple, compound, and complex selectors and selector lists, and use descendant, child, and sibling combinators to target elements for web scraping with precision.

Write a Python scraper that downloads the highest resolution Unsplash images by keyword, excluding premium watermarked ones, using both HTML and API approaches with pagination, saving images locally.

Prospect the site by analyzing image search results, filtering ads and premium images, and inspecting HTML structure to identify reliable selectors; compare HTML parsing with the API for high‑resolution images.

Learn to scrape html with python by building a minimal main.py using httpx and the selectolax html parser to fetch Unsplash search results and extract image urls via css selectors.

We parse the HTML tree to extract images in anchors and figures, then filter URLs to exclude plus, premium, and profile sources.

Extract high-resolution image URLs from image nodes by parsing source sets, filtering unwanted terms, and extracting the base URL through a utility function in a web scraping workflow.

Build a save_images utility that downloads image URLs to a local images directory, prefixes each file with a tag, and saves as jpeg with wb using url-derived filenames.

Learn to add robust logging to a Python web scraping project using the standard logging module, configure levels and formats, and track download progress, saving status, and image sizes.

Explore calling the Unsplash API with Python, fetch images via get requests, parse photos results to extract full-resolution image URLs, and handle pagination for future downloads.

Learn to extract canonical image URLs from a JSON API using Python by navigating results, taking the raw URLs, removing query parameters, and filtering non-premium images.

Learn how to paginate with api requests, constructing urls with page and per page, filtering by photos, and looping through pages to collect a target number of images.

Define a download utility to fetch image URLs from a keyword, save them to a destination directory, and track successful downloads while looping through pages until reaching the target number.

Explain how JavaScript dynamically renders web pages, showing why initial html differs from the html rendered in the browser and how this affects scraping with requests and css selectors.

Discover Playwright as a headless browser for web scraping, compare it with Selenium, and learn its cross-browser capabilities, intuitive API, and suitability for automation and web testing.

Learn to render a javascript-heavy site with playwright and select legs, install chromium, and extract the budget value by waiting for network idle and parsing the html content.