Web Scraping 101 with Python3 using REQUESTS, LXML & SPLASH

Learn how HTML converts into an element tree via the ElementTree object, with tags like p and a forming a hierarchical structure that mirrors the HTML document in Python.

Extract titles and list items from HTML with an element tree, using find and find_all, navigate paths like head/title and body/li, and clean text with strip.

Discover how to select and extract from HTML with XPath in lxml, replacing find with XPath, retrieving text with text() and //tag/text(), and cleaning results with map, strip, and join.

Learn to use css selectors with lxml to extract html elements for web scraping, including installing cssselect, converting the tree to an html element, and retrieving titles and links.

Explore XPath and CSS selectors for querying XML and HTML documents, covering tag, attribute, and position-based selections, with practical comparisons and downloadable cheat sheets.

Learn the fundamentals of CSS selectors, including tag, class, id, and attribute selectors such as data-identifier, and patterns for starts-with, ends-with, contains, plus descendant, direct child, adjacent, and nth-child selectors.

Master CSS selectors for web scraping: target tags, classes, IDs, attribute selectors, and substring matches, and understand combinators like space, >, +, and ~, plus nth-child.

Learn XPath fundamentals for web scraping, compare XPath with CSS selectors, and practice selecting elements, attributes, and text using contains, starts-with, ends-with, text(), and last().

Learn how to navigate the HTML tree with XPath, using the parent, ancestor, preceding, and preceding-sibling axes, plus the node() function for dynamic upward selection.

Navigate down the html tree with xpath, using child, following, following-sibling, descendant axes, and the node() function to select elements and their descendants for web scraping.

Master XPath: select elements with //div, filter by [@id], use position() with or and, contains; ends-with isn’t supported in XPath 1.0, and navigate axes like parent, ancestor, and child.

Analyze how the client and server exchange http requests and responses, focusing on get and post, to render web pages. Gain a view of the web workflow before scraping sites.

Learn how to fetch HTML markup with Python requests, inspect responses with status codes like 200 and 404, and choose between resp.text and resp.content for web scraping using LXML later.

Explore the differences between browser requests and Python requests, examine request and response headers with dev tools, and learn to set a user agent to avoid being blocked when scraping.

Identify and extract the book title, price, stock remaining, and product description from a mock website page using XPath expressions, browser inspection tools, and techniques like contains and following-sibling.

Build a Python3 web scraper by converting the response to an element, applying XPath to extract title, price, availability, and description, and package the data in a dictionary.

Learn two practical data cleaning approaches for web scraping: use regex to extract digits, and use Python functions with isdigit, filter, and join to obtain the stock.

Define Python functions to write scraped data to JSON and CSV files. Use json.dump to serialize dictionaries and csv.DictWriter with headers to create clean CSV output.

Learn to turn a scraping project into a command line app using Kleck and click decorators, with options for book and file name, default values, and auto-generated help.

Learn to scrape a cryptocurrency site with python3 using requests, extracting name, market cap, price, and 24-hour change, and disable JavaScript to view the static HTML before using Splash.

Scrape currency data by building XPath expressions to extract currency name, market cap, price, and percent change, handling multiple currencies and pagination with requests, lxml, and splash.

Master recursive pagination by detecting the next button with XPath, looping pages via requests, joining absolute URLs, handling errors, and saving scraped data to MongoDB.

Store scraped data in a MongoDB cloud database using Python and PyMongo; configure cluster, create a user, whitelist IPs, then insert data into currencies database.

Prevent duplicate records in mongodb by using the coin number as the id and updating entries when price, market cap, or 24-hour change differs.

Update XPath expressions to adapt to CoinMarketCap changes, disable JavaScript for testing, and extract coin number, name, market cap, price, and 24h change.

Explore the differences between api and html scraping by inspecting network requests, disabling javascript, and extracting product data from a site using xhr, post, and json payloads.

Learn to test and generate code for web APIs with Postman, including setting headers, choosing get or post, handling responses, and exporting Python requests code for automated scraping workflows.

Parse APIs with Python by retrieving JSON responses, iterating over products, and extracting id, display name, price, size, unit, and image to build absolute URLs.

Leverage recursion to scrap remaining products across pages by turning the code into a swapper function with a single page number argument defaulting to one, and a stop condition.

Test API behavior with an invalid page number to establish a stopping condition. Wrap the code in a try-except to return none on missing data during recursive scraping.

Learn to store products in a secure lightweight sqlite3 database by creating Walgreens.db, building a products table with a primary key, and inserting records with error handling.

Explore how Splash renders JavaScript-heavy pages with requests and how to write scripts and select elements. Grasp why requests cannot handle JavaScript and how Splash acts as a browser engine.

Set up splash using DOCA on a PC by configuring virtualization tools, disabling Hyper-V on Windows, installing Docker Toolbox, VirtualBox, and creating a fixed port for reliable access.

Explore Splash's user interface, render a webpage to capture a screenshot and the estimated markup, using a scripting language and wait logic to ensure resources load.

Automate Google searches with splash by locating the input with css selectors, filling it, and submitting via enter or clicking the search button, while adjusting the viewport for full results.

Explore Splash history to inspect request headers and cookies, and inject custom headers and user agents using set user agent, set custom headers, and on request callbacks for web scraping.

Learn how to configure splash for web scraping with private mode, cookies, custom timeouts, and user agents; inject headers, disable images, and render pages for reliable data extraction.

Learn to use Splash with Requests by posting a Lua script to a Splash container, then fetch the rendered markup for scraping.

Parse the page to extract each product's name, original price, and discounted price using xpath targeting the goods item container. Clean whitespace and export results as json or csv.

Learn how Twitter authentication works for web scraping with requests, covering login redirects, session requests, referrer headers, and dynamic authenticity tokens.

Learn to simulate login to Twitter with Python requests by creating a session, handling headers and user agent, extracting authenticity tokens, and posting login data to sessions.

Learn to parse and clean HTML to extract a tweet’s text from a profile page using XPath, headers, and requests, then remove markup with a cleaning library.