5 H Data Challenge Course - Scraping-Cleaning-Analysis

Install and use Beautiful Soup and the requests library with pip on Windows, Mac, and Linux. Guide Python beginners through setup and basic troubleshooting.

Install and launch the Jupyter notebook, then install requests, pandas, and SQLAlchemy to prepare data scraping, cleaning, and output to a PostgreSQL database.

Install PostgreSQL on Windows from the official site, select the correct options, and set a password to enable the database and pgAdmin access.

Create ten city datasets by scraping real estate listings and extracting links and data such as address, bedrooms, bathrooms, square feet, year built, parking, and price across multiple pages.

Kick off the scraping workflow by importing beautiful soup and requests, fetch the first page, and extract result links. Filter for the data-test-id attribute to obtain the valid 40 results.

Extract result links from a scraping page. Concatenate protocol, homepage, and relative paths to form absolute urls, then prepare to collect address, price, beds, baths, area, and parking data.

Extract data from the first link by fetching the page, parsing with BeautifulSoup, and retrieving address, bedrooms, bathrooms, square feet, year built, parking, and price.

Demonstrates looping through all links on the first page to scrape and collect property data—address, bedrooms, bathrooms, area, year built, parking, and price—into a dataframe.

Scrape data from multiple pages by iterating page numbers and constructing urls, then compile results into a data frame.

Merge ten datasets and standardize the data frame through a seven-step cleaning process, including index resets, missing data handling, and computing price per square foot for a PostgreSQL-ready database.

Import modules and access the current directory with os, list and merge ten datasets using pandas, read Excel files, fix openpyxl errors, and create a combined data frame for cleaning.

Inspect the combined data frame for missing data, visualize gaps with a seaborn heat map, then drop incomplete rows, remove duplicates, and reset the index for a clean dataset.

Rename the price column to price in dollars, strip the dollar sign and commas, and convert the column values to integers to make the data database readable.

Rename the area column to area in square feet, remove the square feet label and extra spaces, and use replace to remove commas for clean numeric values.

Clean the bedrooms and bathrooms columns by removing bad string elements, converting studio entries to one bedroom, and normalizing values for accurate counts and analysis.

Modify the parking column by applying a keyword-based lambda function in Python to label entries as yes or no, using garage, Copart, car, or open as indicators.

Convert bedrooms, bathrooms, and area square feet to integers by checking string or numeric types and casting, with year built and price handled in a separate video.

Convert the year built column to an integer by validating numeric values, handling missing data, and replacing no info with zero, then verify the column becomes numeric.

Convert the price column to integers by removing plus signs with a lambda, override the old column, verify numeric dtype, and prepare to compute price per square foot.

Create a new price per square feet column by dividing price by area square feet and rounding to two decimals, then address outliers.

Explore the data analysis section by applying cleaned data to answer ten questions using Penas library or sequel, or visualization libraries, with prerequisites installing PostgreSQL and configuring inputs and outputs.

Import and prepare real estate data by loading Excel into a pandas dataframe, cleaning columns (area in square feet, price in dollar), and exporting to PostgreSQL via a SQLAlchemy engine.

Answer question one by counting results per location with pandas value_counts and compare with a PostgreSQL group by, then visualize with Matplotlib.

Compute the minimum, maximum, and average price across observed locations using the panel's library, pandas, and SQL. Indianapolis is the minimum and New York is the maximum, with formatted results.

Group by location to compute the median price per square foot, store results, sort in descending order, and validate with SQL queries and a plot of the real estate table.

Target the real estate data frame to filter San Francisco and retrieve the price per square foot, identifying the maximum value.

This seventh data analysis video answers question five by computing the lowest price per square foot in Miami. It mirrors the prior San Francisco high-price query.

Compute mean house price and mean price per square foot by location using group by, then sort and graph results; New York is highest, Indianapolis is lowest.

Find the most expensive Las Vegas house using the IDEX max index in the data frame, then extract seven bedrooms and seven bathrooms, and cross-check with PostgreSQL.

Identify the cheapest San Diego house in a real estate data frame with the index min function and confirm if it has a parking opportunity via code and sql cross-checks.

Compute the median house price and median price per square feet, then plot both on a single graph with two y-axes to reveal their correlation.

Analyze pivot tables by location and parking to compare mean price per square foot. Results show no consistent correlation between parking availability and price per square foot.