Complete Python Web Scraping : Real Projects & Modern Tools

Learn to use beautiful soup and requests to scrape books from books.toscrape.com, install libraries, make requests, use selectors, paginate, save data, download images, and employ proxies.

Install requests and bs4, then make the first web request to books.toscrape.com. Parse HTML with BeautifulSoup, view the response using prettify, and set the stage for learning selectors.

Learn to use Beautiful Soup selectors to extract text and links by class name, attributes, and text, and navigate the DOM with parent, siblings, and children.

learn to scrape a single book's data from books.toscrape.com using Python requests and BeautifulSoup, extracting name, price, category, stars, upc, availability, stock, and image link.

Master pagination in Python web scraping by using a known page count with a for loop or an unknown count with a while loop, and extract books from article.product_pod.

Constructs an inner loop to iterate over the books list, extracts each book URL, and scrapes data from every book using headers with a user agent for requests.

Save scraped data in a dictionary, convert to a pandas data frame, and export to book.xlsx with openpyxl while scraping 50 pages from booktoscrape.

Download images with the requests library by reading image URLs from a text file and saving them to a book_images folder as 1.jpg through 1000.jpg.

Learn to use proxies with the requests library, including unauthenticated and authenticated proxies, by loading proxy list, setting a user agent, handling errors with try/except, and testing access to books.toscrape.com.

Explore beautiful soup and the request section through two videos: first explain the task and prompt you to solve it solo, then solve it together.

Practice web scraping by extracting quotes, authors, and tags from quotes.toscrape.com across 10 pages, printing the first quote and its author, and iterating over all quote elements.

Scrape quotes, authors, and tags across pages using Beautiful Soup and requests, selecting elements by class names and handling pagination.

Master web scraping with beautiful soup and requests by building a real life laptop data scraper from eBay, selecting selectors, handling pages, and saving the scraped data.

Learn practical web scraping with Python using requests and BeautifulSoup to extract a laptop's name, price, shipping, and product link, including handling headers and cookies for reliable results.

Use a while loop to paginate in Python by incrementing the page number in the URL, and break when the next button is inactive while handling non-200 responses with continue.

Save scraped eBay data by building a dictionary of name, price, shipping, and product link, append items, and export a pandas dataframe to laptops.xlsx using openpyxl.

Learn basics of selenium, install the chrome driver, use css selectors and XPath, automate login, scrape the first page, loop through pages, and explore infinite scroll, waits, sleep, and actions.

Install selenium and chromedriver by downloading chromedriver for your OS, then add its path to system variables. Install selenium in PyCharm and import webdriver to prepare a browser driver.

Set up a selenium chrome driver with options to start maximized and detach browser, instantiate the driver with these options, and open quotes.toscrape.com/js/ using driver.get to prepare for css selectors.

Learn to use css selectors with selenium to locate page elements by tag, class, id, and attributes, and extract text and attributes such as price, src, and alt.

Learn xpath in selenium using python to locate elements by tag, class, id, and attributes, and extract text, href, or image src across nested and sibling structures.

Automate website login with Selenium by locating the login button using XPath or CSS selectors, inputting credentials, and logging in; then convert the steps into a reusable login function.

Learn to scrape the first page of quotes after logging in, using selector-based extraction with a driver to collect quotes, authors, and tags, and implement a scrape_page function for pagination.

Learn to browse pages with Selenium by detecting the next button in a while loop, breaking on the last page, scraping quotes, authors, and tags to an excel file.

Learn to build an infinite scroll workflow with Selenium to load all items on a page, using Python, WebDriver, and a loop that scrolls to bottom and waits between loads.

Learn how to manage waits in selenium to load page elements, using explicit waits for presence of an element located, implicit waits, and sleep for reliable web scraping.

Learn how to use Selenium action chains to move to an element and perform clicks, troubleshoot click errors, and apply the technique to navigate pages like a next button.

Explore selenium in action with a quotes.toscrape.com exercise, delivered in two videos that explain the task and then solve it together for Python web scraping.

Learn to use selenium to scrape author names, birth dates, and birthplaces from pages, handle pagination, store them in a dictionary, and export to an Excel file with pandas.

Learn to automate web scraping with selenium to login, paginate through author pages, and extract name, birth date, and birth place, then save results to an Excel file using pandas.

Automate IMDb data extraction with Selenium by opening IMDb.com, navigating to the target page, scrolling to load all items, then scraping and saving the data.

set up the project by installing selenium, pandas, and openpyxl; configure chrome options to start maximized with detach, create a driver with implicit wait, and open imdb.com to begin automation.

Adapt the web scraping automation to a changed site structure by updating selectors, navigation steps, and data test IDs, while preserving workflow and cookies handling across pages.

Automate browser actions with Selenium to reach the Oscar-nominated comedy movie list, handling cookies, navigating genres, and applying filters for scraping 1,047 titles.

Write a while true loop to scroll to the end and click the see more button until all items load, using a CSS selector and a two-second wait.

Practice selenium by scraping IMDb director information in a hands-on exercise. In the first video, learn the tasks; in the second, solve it together.

Learn to automate web scraping with selenium to extract directors' heights from a drama director list, load items, visit director pages, and save results to an excel file with pandas.

Explore selenium-based web scraping in Python: navigate pages, handle cookies, apply filters, load all items, and extract director names with heights, exporting to Excel.

Master scrapy basics by installing scrapy, creating a project and spider, using the shell and selectors, parsing books data from books toscrape.com, and exploring pipelines, middlewares, and crawl spiders.

Install scrapy, create a basics project with items, middlewares, pipelines, and settings, generate a book spider to scrape books.toscrape.com, then explore the scrapy shell and css selectors and xpath.

Construct a parse method in a Scrapy spider to extract book links, manage pagination with a next page link, and route book data scraping to the parse_books callback.

Define a Scrapy item in items.py with fields like name, price, category, stars, and image URL, create a book item, and yield scraped data in parse_books.

develop a parse_books method to create and populate a book item from scraped data—name, price excluding tax, upc, availability, category, rating, image URL—yielding items for json export with scrapy.

Scrapy pipelines process scraped items before saving, adjusting fields with an item adapter to uppercase names, extract availability, and convert pounds to dollars, then save to json.

Learn to drop items with scrapy dropper pipeline by checking stock, converting stock to int, and saving only books with at least 10 in the output json.

Create an excel pipeline to save scraped data to books.xlsx, while also exporting a json file, by building an openpyxl workbook, naming the sheet, and writing item fields in order.

learn how to save scraped data to a sqlite database using pipelines, creating the database and booktable, and inserting items with a cursor then committing the connection.

Explore scrapy middlewares that rotate user agents and proxies, manage requests and responses, using settings.py lists, and retry on non-200 responses.

Learn to build a Scrapy crawlspider with link extractors and rules to navigate catalog pages, category pages, and book pages, then scrape data into JSON.

Enter scrapy section 2 and tackle an exercise on quotes.toscrape.com, where you will try solving it solo before we review together in the next video.

Build a scrapy exercise that creates a quotespider for quotes.toscrape.com and parses each quote's text, author, and tags. Follow pagination with response.follow and export results to json.

Build a Scrapy project named quotes, create a quotespider, and define a quote item with quote, author, and tags. Save the results to quotes.json.