Web Scraping and API Fundamentals in Python

Install Anaconda, Python 3, Jupyter Notebook, and the relevant packages to set up the environment for Python-based web scraping.

Explore why Python and Jupyter underpin data science, highlighting open source, cross-platform availability, rich packages, IPython notebooks with kernels, and how Jupyter enables collaboration.

Install Anaconda to get Python, Jupyter Notebook, and data science packages, choosing Windows (64-bit) with Python 3, and complete the standard installer to launch the Jupyter dashboard.

Navigate the Jupyter dashboard to manage files and folders with checkboxes, rename or delete items, upload notebooks, and create ipynb notebooks with an interactive shell.

Navigate the Jupyter notebook interface, use code cells and markdown cells, execute with ctrl+enter or shift+enter, and master keyboard shortcuts like cut, copy, paste to run cells efficiently.

Learn which libraries come with Anaconda for web scraping and APIs, including numpy, scipy, pandas, requests, and Beautiful Soup, and how to install requests-html with pip.

Discover how web based APIs define a contract between client and server, using http requests and json, with public examples, documentation, and concepts like pagination and registration.

Explore how http requests fetch web content, comparing get and post methods, and understand status codes like 200 and 404 and json data in web APIs.

Learn how JSON serves as a language-independent, human- and machine-readable data exchange format for APIs, built on dictionaries and lists, with examples of country data and nested structures.

Explore a currency exchange rates API by making a get request with requests in Python, parse JSON with response.json(), and inspect keys like base, date, and rates.

Learn to specify get request parameters, including base and symbols, with a question mark, ampersand, and equals sign to fetch targeted exchange rates using the API's JSON response.

Explore additional api functionalities by using historical and latest endpoints, building start at and end at dates, and filtering for currencies like GBP, then handle json dumps and errors.

Build a python currency converter using the exchange rates API, collecting date, base and target currencies, and quantity, then fetch rates, compute the result, and display it with error handling.

Explore iTunes search API in Python by building requests with term and country using the requests library, handling spaces, and inspecting JSON response keys like result count and results.

Convert iTunes API results into a pandas data frame and export to csv or xls, using the data frame constructor and a simple workflow, with pagination next time.

Explore API pagination using the GitHub Jobs API, showing how to fetch 50 results per page, loop through pages with the page parameter, and aggregate results with the extend function.

Learn to authenticate with the edamame nutritional analysis API by obtaining an id and key and using them in the endpoint, noting free access and 200 requests per month.

Submit a post request to the Edamam API with a json body containing title and engr and a content-type header; inspect the response and export nutrients to csv.

Learn the basics of HTML, its structure and syntax, encoding, and how browsers render HTML and provide dev tools for inspecting and scraping web pages.

Explore the basics of HTML syntax, including the HTML element, opening and closing tags, doctype HTML, and the head and body structure, with examples of paragraphs (p) and headings (h1–h6).

Learn how tag attributes use name-value pairs to control alignment, links, and element behavior, and examine class and id attributes for precise scraping.

Learn to identify common HTML head and body tags, metadata usage, and data-structuring elements like div, span, iframe, images, lists, and tables, plus basic CSS and JavaScript concepts.

Learn how CSS describes the visual presentation of HTML, including inline, internal, and external styles with selectors, and how JavaScript enables interactivity that impacts web scraping.

Master character encoding from ASCII to Unicode and UTF-8, and learn HTML symbol encoding via named entities and decimal or hexadecimal references, including euro sign and non-breaking space.

Explore XHTML and code style, noting that HTML rules are guidelines and sloppy markup can affect scraping, especially on smaller sites, where strict syntax matters.

Learn to use the Beautiful Soup Python package to extract data from HTML, demonstrate real use cases, and scrape information from websites beyond official APIs.

Discover the five-step workflow of web scraping: inspect the page, fetch HTML with a GET request, choose a parser, parse, and build a BeautifulSoup object to navigate the parse tree.

Practice hands-on web scraping in Python by fetching a Wikipedia music page with requests, building a BeautifulSoup object with a parser, and exporting the HTML to a file.

Master searching the HTML tree by attributes with Beautiful Soup, using find and find_all to filter by id, class, and href, or by an attributes dictionary for robust element selection.

Extract data from the HTML tree with Beautiful Soup by reading a tag's name via attribute dot name, accessing attributes (href, class) via dictionary-like indexing, and using get for defaults.

Explore extracting text from html tags with BeautifulSoup using the dot string and text attributes, handle nested tags, navigate to parent elements, and iterate strings with stripped_strings for clean text.

Explore hands-on web scraping with Beautiful Soup by extracting and normalizing links, converting relative URLs to absolute using urljoin, and filtering internal links with list comprehensions.

Identify div tags with a role attribute to extract links labeled as main article or see also, then loop through them to build a list of urls using url join.

Learn to scrape multiple pages automatically by extracting main text from paragraph tags. Loop through each page in Python, handle errors, and build a url-to-text dictionary for clean, usable data.

Use requests and beautiful soup to extract movie data from a rotten tomatoes list, including title, year, score, critics consensus, cast, and director. Switch to the lxml parser for speed.

Learn to extract movie titles, years, and scores by inspecting the page structure with dev tools and parsing data via Beautiful Soup using list comprehensions.

Learn to scrape critics consensus, cast, and director data from movie pages by parsing HTML with class-based divs, removing the leading phrase, and handling missing director links.

Extract cast information from movie pages by locating the div class cast, retrieving names through links, and joining them into a single string, then store in a simple structure.

Store scraped movie data in a structured form with a Pandas data frame, display it fully with set_option, and export to CSV or Excel for easy sharing and analysis.

Discover how to scrape an html table with BeautifulSoup and pandas read_html in Python, extracting the capitals column from a Wikipedia page and using pandas to simplify table handling.

Identify common roadblocks in web scraping and apply practical fixes using request headers, user agents, and cookies. Manage login, sessions, rate limits, and JavaScript generated data.

Explore the requests-html package as a single, simplified alternative to the Beautiful Soup and requests combo, with full JavaScript support to render dynamic pages.

explore the requests-html package by using an HTML session to fetch and parse a page, extract links with r.html.find, convert to absolute URLs, and apply CSS selectors for data scraping.

Learn to search text between phrases with the search and search all methods in requests HTML, handling patterns, placeholders, and raw HTML.

Learn how CSS selectors filter and select HTML elements for web scraping in Python, using tag, id, class, and attribute selectors, plus combining and nesting techniques.

Learn to render JavaScript with the render method in an async HTML session to fetch dynamic content, then compare pre- and post-render HTML for divs and links.