Scrapy MASTERY Course - Become a Python Web Scraping MACHINE

Learn to define web scraping, distinguish crawling from single-page scraping, and apply data detective work to extract diverse data types—from HTML to JSON—using automated bots.

Understand how the internet works across its layers to diagnose web scraping issues, perform Python requests, and build your first scraper.

Install scrapy with pip, start a new project, and create a spider for pirate.scraping.com; use XPath in parse to extract the title into a dict, then run the spider.

Learn to extract data from scrapy response objects using XPath, compare with CSS selectors, navigate with //, and extract inner text with text().

Navigate a page’s html to locate clean and hidden data, and extract fields such as air FC number, title, and description using xpath text and the at sign.

Learn to build a Wikipedia article crawler with Scrapy by extending CrawlSpider, using LinkExtractor and Rule to follow wikipedia.org/wiki links, and extract title, URL, and last edited date.

Learn to collect and store scraped data with Scrapy by defining an article class in items.py with title, url, and last update, and write results to csv, json, or xml.

Configure Scrapy with the settings.py file to permanently cap page scraping at ten. Define global and per-spider output formats (csv, json, xml) and write data to files via command line.

Configure Scrapy settings and spiders to extract data, then demonstrate item pipelines: check item pipeline and cleaned pipeline in pipelines.py to validate, clean, deduplicate, and store by attributes.

Build a complete Scrapy project with multiple spiders, a single item class and pipeline, and export data to JSON by scraping news articles from Associated Press, CNN, and Yale News.

Celebrate the halfway point in the scrapy master course and share feedback on what you learned so far to improve the course and expand your web scraping skills.

Scrape news from the Associated Press, CNN, and Yahoo News to build a complete news article item, extracting metadata, description, date, author, and full text using XPath, JSON, and Python.

Learn how to scrape forms with Scrapy by sending get and post requests, generating start URLs, and extracting content with XPath using form data and parameters.

Discover how dynamic content loads via hidden APIs and network requests, inspect the URL calls, and modify parameters to access data unseen on the page.

Explore how robots.txt and the robots exclusion standard guide web crawlers, and leverage Scrapy's sitemap spiders to crawl XML site maps from sites like CNN for organized article access.

Assess a CNN articles scraping project by quickly using the CNN sitemap to estimate available articles over ten years, plan a scalable scraper, and outline key findings for the client.

Use robots.txt to locate site maps, extract the article sitemap, and build a quick crawler to count URL tags, estimate 253,589 articles from January 2011 to present, exporting to csv.

Explore a login page, observe the post to welcome.php with username iframe and login equals one, then override make_request_from_url to inject cookies in Scrapy and verify login.

Master browser automation with Selenium in a Scrapy project by wiring Chrome driver, Scrapy Selenium middleware, and a Selenium request to handle dynamic pages and wait time controls.

Learn to interact with a page by logging in with a username and password, using a selenium driver to navigate, click login, and extract profile text with Scrapy and XPath.

Choose a project you genuinely care about and learn by doing, while building MySQL, Python data analysis, and web scraping data structures.