Web Scraping APIs for Data Science 2021 | PostgreSQL+Excel

Install essential libraries for web scraping by using pip to install Jupiter notebook, requests, pandas, and SQLAlchemy, then launch Jupiter notebook to create Python notebooks and work with PostgreSQL.

Learn the fundamentals of api scraping, inspect network requests to identify json data, and extract book details (title, subtitle, author, publisher, year, price) in python for excel.

Explore the fundamentals of API scraping and complete a beginner project. Loop through ten books, extract title, subtitle, author, publisher, publication year, and price, and export to Excel.

Extract six data points from the search_results JSON: model, mileage, year, dealer name, and price, starting with the first result and looping through all 20 cars via API scraping.

Loop through 20 results on a single page to extract model, mileage, year, dealer name, and price into a data frame and export to Excel.

Learn to scrape data across five pages to collect 100 results using Python requests and JSON parsing, extract fields into a dataframe, and export to Excel.

Explore how the art gallery site's infinite scroll uses its API, inspect network requests, and use Python requests in a notebook to paginate (0, 48, 96) and extract artist data.

Learn to extract nine data points from a nested api response, including name, first name, last name, user name, country, and views, by traversing keys and indices for 48 results.

Extract data from a page by looping 48 results, creating nine lists, and appending name, first name, last name, user name, country, views, appreciations, comments, and url; build a dataframe.

Learn to paginate across ten pages to collect 480 results, iterating page parameters and building a data frame of nine data points, then export to Excel.

Advance your data scraping by collecting 10,000 used car listings from TrueCar, first in New York, then San Francisco, aggregating into data frames and loading into PostgreSQL.

Explore scraping New York used car data by inspecting the site's API, extracting the second-page request, converting it to Python, and iterating page parameters to build a JSON results set.

Extend a web scraping workflow to loop through 30 results on the first page, collect brand, model, mileage, year, and price, and scale toward 5000 results with a pandas dataframe.

Learn to crawl and aggregate thousands of web results by paginating requests, building JSON objects, and exporting data to Excel and PostgreSQL for San Francisco datasets.

Extract 5000 used car results for San Francisco using pagination in a Python notebook. Prepare a data frame and request workflow for PostgreSQL export in the next video.

Combine dataframes with pandas concat, ignore the index, and convert a price column from float to integer. Save the merged data to Excel and prep for PostgreSQL export.

Install PostgreSQL on Windows by downloading the official installer, selecting options, setting a password, and launching the database interface to run queries with your data.

Connect a Jupyter notebook to PostgreSQL using SQLAlchemy, create an engine, and run SQL queries on cars_results to filter Jeep 2015, Lexus under 30000 miles, and Mercedes-Benz under 45000.