Web Scraping in Python With BeautifulSoup and Selenium

Learn to fetch a web page's HTML in Python using requests and BeautifulSoup, inspect the page, and parse with lxml to prepare data for scraping.

Learn to use BeautifulSoup's find function for web scraping to filter HTML by tag and attributes, extract headers and prices, and understand when to use find_all for multiple matches.

Learn how to extract data from nested HTML by narrowing to a single box using BeautifulSoup, then apply find_all and find to retrieve product names, prices, reviews, and descriptions.

Learn to scrape stock data from MarketWatch using Python by importing HTML, and extract the current price, closing price, 52-week range, and the analysis rating as strings.

Master web scraping in Python with BeautifulSoup and Selenium to extract stock data—price, closing price, 52-week range, and analyst rating—using nested HTML techniques in hands-on coding exercises.

Learn to scrape a web table using Python with requests, BeautifulSoup and pandas, converting the Worldometers population table into a ready-to-export data frame for Excel.

Learn how to locate a specific table in a web page’s HTML using BeautifulSoup, extract column headers from th tags, and create a data frame using those headers.

Extract table data by identifying tr rows and td cells, skip the header, iterate over rows, collect text, populate a dataframe, and export to csv for easy analysis.

Learn to scrape a web page with Python using requests and BeautifulSoup to extract a table by its summary attribute, parse headers and rows, and build a pandas dataframe.

Learn to scrape airbnb.com using Python with BeautifulSoup and Selenium, analyzing code to extract title, price, and description while adapting to site updates and avoiding copied code.

Learn to scrape multiple pages from a single website using Python, BeautifulSoup, and requests, extracting titles, descriptions, amenities, price, and reviews across Airbnb listings.

Explore multi-page scraping with BeautifulSoup by locating the next page URL from an aria-label next link on Airbnb listings, building the full URL, and looping through pages 1–15.

Continue scraping across 15 pages to extract posting data—title, price, rating, description, and email—and compile them into a table while identifying the lowest price posting.

Learn to scrape multi-page listings with BeautifulSoup and Selenium, extracting price, title, links, and rating, handling missing data with try/except, and compiling results into a pandas dataframe and csv.

learn to scrape car listings with python using requests, BeautifulSoup, and pandas to extract link, name, price, and color, store results in a dataframe, and paginate across ten pages.

Learn how JavaScript driven websites defy simple scraping with requests and BeautifulSoup and why Selenium is needed to load dynamic content, click buttons, and extract shoe names and prices.

Selenium is a browser automation tool that controls a browser to load pages and wait for JavaScript. It captures html, supports clicking, typing, waits, and scrolling for JavaScript-driven pages.

Install selenium with pip in the Anaconda prompt and download the Chrome driver that matches your Chrome version, then configure the webdriver and test opening a page.

Discover how to use selenium's web driver to load pages, view html, and locate elements by id, name, tag, and xpath, enabling visual web scraping with driver.get and chrome driver.

Learn how to use XPath to locate web elements with Selenium, extract prices and titles, and click the see more button by building robust path expressions.

learn why selenium's find element methods often fail in favor of XPath, and how to extract shoe prices by iterating the XPath index to pull multiple prices.

Learn to use selenium to click buttons on web pages for web scraping by locating elements with XPath, then drive navigation from a Google search to Wikipedia pages.

Learn to implement self scrolling in selenium with Python to load more images by scrolling to the bottom using window.scrollTo and document.body.scrollHeight.

Demonstrates building a Selenium web scraping workflow to search Google, open IMDb, and capture page and poster screenshots. Applies input handling, XPath selection, waits, and scrolling.

Learn to implement infinite scrolling with selenium and beautiful soup to load all products, scrolling to the bottom until the page height stabilizes, and structure data with pandas.

Combine selenium-driven infinite scrolling with BeautifulSoup to extract product details from a page. Build a dataframe of links, names, subtitles, full price, and sale price with basic error handling.

Demonstrate end-to-end web scraping with infinite scrolling to collect Nike product data, export a csv, and identify the best discounts.

Master scraping all outerwear products from Union Los Angeles using infinite scrolling, extract link, name, and price, then compile a table and identify the most expensive item.

Scrape a page with infinite scrolling using Selenium and BeautifulSoup, load all products, extract link, vendor, title, and price, and save to a pandas dataframe.

Code a Twitter scraping script from scratch in Python using BeautifulSoup and Selenium, logging in, navigating the search tab, and extracting tweets with XPath while handling dynamic HTML changes.

Master Twitter login automation using Selenium WebDriver, locating inputs via XPath, sending credentials, clicking login, and applying explicit waits to load pages.

Leverage selenium to search Twitter by locating the search box with xpath, sending keys the celebrity name, and pressing enter, then navigate to the people tab and open the profile.

Master scraping tweets from a profile using BeautifulSoup and Selenium; extract tweet text, manage dynamic loading by scrolling, deduplicate, and collect hundreds of posts.

Practice a web scraping exercise on Indeed with Python, BeautifulSoup, and Selenium to input a job title and location, then extract links, titles, companies, locations, salaries, and dates across pages.

Learn to scrape job postings from Indeed using Python with Selenium and BeautifulSoup, extract title, company, location, salary, and date into a pandas DataFrame across multiple pages.