Becoming a Data Analyst Using Python, from scratch

Learn to start data analysis by importing the pandas library as PD, then read a csv dataset from a web address and store it in a local variable called first_data_set.

Download the dataset and store it as a local variable. Use pandas head to view the first five rows and dtypes to identify int, float, bool, and object columns.

Inspect a Chicago crimes dataset with head, tail, and dtypes, then rename it to Chicago crimes and copy it with pandas.

Use the pandas info function to inspect a dataset’s structure, data types, and non-null counts, revealing 22 columns, 238,317 rows, and where missing values occur.

Learn to count null values in a pandas data frame by using isna and sum, compare non-null counts with totals, and identify which columns contain missing data.

Explore selecting specific rows in pandas with the dot loc tool, using zero-based indexing to fetch a single row or a range, and understand output orientation.

Zoom into a specific column by taking a subset with square brackets in Python, displaying only that column, or use an array for multiple columns in pandas.

Learn to combine column subsetting with dot.loc to view specific rows and columns, count null values with isna and sum, and apply this subset as a standalone data frame.

Learn to remove rows and columns in pandas using drop, with inplace or a new variable, verify with head, and reload data with read_csv on the Chicago crimes dataset.

Learn to filter the dataset by a single condition in Python, selecting rows where the primary type equals theft in the Chicago crimes dataset, and understand true/false results.

Learn to create and save data subsets by condition, using correct boolean and object type rules for fields like theft, arrest, location description, and community area.

learn how to combine two or more conditions using the and operator in Python to filter data, such as theft records with arrest true in the Chicago crimes dataset.

Explore how the or operator evaluates two conditions and returns true when at least one is true, using arrest and domestic criteria in a Chicago crimes dataset.

Learn to filter data using greater than or equal to and less than or equal to conditions, combine criteria with and, and apply comparisons to numeric columns.

Learn to return records not equal to a value in Python using the exclamation mark, illustrated by Chicago crimes community area not equal to 23.

Use the is in function to check whether the primary type is in a given array of values, such as theft, battery, and assault, making multi-condition filtering concise.

Apply the isna() method to filter records with null values. Use notna() to list non-null values and filter location description in the Chicago crimes dataset.

Apply summary statistics to reveal the distribution of values in a dataset column, building on data reading, null checks, and filtering, and prepare for the next dataset analysis.

Learn to read an incomes dataset with pandas read_csv, fix header issues with header=None, rename columns with rename and inplace, and check for missing values to prepare data for analysis.

Explore the data set to compute central tendency with the mean for numeric columns in pandas, starting with hours per week and then combining hours and age to compare averages.

The lecture explains that the mean alone can't capture central tendency and variation, and shows using min, max, and the median to interpret data.

Demonstrate using the quantile function to compute mean, median, and the 25th and 75th percentiles, plus 20th and 99th percentiles, to understand a numeric variable's distribution.

Visualize the distribution with a box plot, showing the median, the 25th and 75th percentiles, the interquartile range, and potential outliers, using pandas boxplot on the age column.

Explore how histograms visualize the distribution of numeric variables by age and hours per week, showing frequency across ranges. Compare with box plots to spot concentration and outliers.

Discover how the describe() function quickly computes key statistics, including the mean, standard deviation, min, max, and quartiles, for all numeric columns, providing a readable summary of a dataset.

Explore non numeric columns by identifying unique values and computing value counts and percentages, using examples like sex and race to reveal frequencies.

Combine two non-numeric columns, such as race and sex, using value_counts with double brackets to compute counts and percentages for each group. Normalize to display proportions.

Learn to compute frequencies with the group by function, using size to count groups by one or more columns — such as sex and race — to reveal group sizes.

Explore the describe function's handling of numeric and non-numeric columns, using include='all' to reveal unique values and the most frequent value, even when some statistics are not applicable.

Visualize non-numeric variables with pie charts to show group proportions, using value_counts with a plot or using groupby().size() with a plot; the groupby method is emphasized.

Master adding number labels to pie charts using the auto pct option in pandas, control decimals, and compare gender and race distributions to better understand data.

Visualize non-numeric variables with bar charts in pandas, grouping by education or race, sizing groups, sorting with sort_values to show distributions from most to least frequent.

Group data by sex to compute per-group means for hours worked per week, then treat the result as a data frame and apply describe to reveal min, max, and percentiles.

Visualize group summary statistics with pandas by plotting box plots grouped by sex and race, comparing medians and percentiles, and rotating x-axis labels for clarity.

Plot box plots of hours per week grouped by multiple columns, such as sex and race, to compare cross-group differences and education effects using isin.

Visualize group statistics with box plots and education-by-race grouping to compare hours worked per week. Learn filtering, grouping, and distribution analysis to see how education level relates to work hours.

Explore the unstack function to convert grouped means into plottable data for bar charts. Use group by and mean to compare hours per week across sex and marital status.

Use pandas crosstab to create cross tabulations by two variables, counting records across race and education, with the normalize parameter to show proportions.

Demonstrates using box plots to compare hours worked per week across education levels and sex, then uses pandas crosstab to explore relationships among education, marital status, income, and gender.

Use pandas to analyze the Chicago crimes dataset: read_csv dataset, describe the year, group by primary type, and plot a stacked crosstab of arrests versus no arrest to reveal trends.

Explore how to measure relationships between numeric variables using correlation on a car prices dataset, examining engine size, mileage, and price with pandas.

Learn to visualize the correlation matrix of numeric variables using dataframe styling with background gradient and cmap, and explore heatmaps and an upcoming library.

Learn to visualize numeric correlations with Seaborn by plotting heatmaps of a data frame's correlation matrix, importing Seaborn as SB, and exploring cmap options to adjust color scales.

Explore how scatter plots reveal relationships between numeric variables using pandas plotting, plotting price against mileage to reveal an inverse correlation and the idea of a perfect correlation.

Use seaborn scatter plot to visualize the relationship between year, price, and mileage, coloring by car type and sizing points to reveal how different categories relate to price and mileage.

Explore Seaborn's relplot to visualize relationships between two variables, create separate plots by type, and subset data (car prices) for clearer comparisons.

Explore Seaborn's lmplot to visualize best fit lines with scatter plots, using year as x and price as y, and compare slopes across car types and transmissions.

Plot geographic locations with latitude and longitude using Plotly Express scatter mapbox to visualize Chicago crimes and arrests on open street map.

Read the Airbnb dataset with pandas, load the csv, inspect columns, rename the price column, and explore geographic data with seaborn and plotly express to analyze price drivers.

Validate dataset types and integrity, confirm no null values, and inspect structure with info. Summarize statistics across all columns, including mean, median, 75th percentile, city distribution, and price ranges.

Analyze price data with box plots to reveal skewness and outliers, compare top ten listings by city, and quantify city-level frequencies to uncover pricing patterns.

Group the data by city to generate price statistics with describe, then compare room types—entire homes, private rooms, and shared rooms—and visualize distributions with a stacked bar plot.

Explore how Seaborn scatter plots reveal the relationship between price and distance from city center, and to the metro, showing higher prices nearer center and metro with a downward trend.

Calculate price correlations with other variables, subset to price, and sort to reveal weak links; heatmap shows guest satisfaction is not correlated with price and aligns with overall cleanliness.

Plot geographic locations of listings using latitude and longitude with Plotly Express and scatter mapbox, coloring and sizing points by price to reveal city-level patterns on an open street map.

Learn to manipulate datasets by addressing common data preparation issues, applying commands to format data for analysis, and ensuring datasets are ready for analysis.

Explore how to handle null values in datasets by dropping records with missing values or by replacing them with a median or a specific value, using the Chicago crimes example.

Assess how data analysts handle outliers by calculating the interquartile range and the 1.5 times IQR limits to decide when to remove extreme values and refine datasets.

Learn how to fix a column that should be numeric but is stored as text by converting with pandas astype and to_numeric. Use errors='coerce' to handle non-convertible values.

Learn how to clean data by removing leading and trailing spaces from column values using the STR attribute and the strip function, ensuring accurate filtering and analysis.

Learn to convert a string column to numeric with to_numeric, handling errors. Remove commas using STR.replace by replacing them with an empty string to ensure full conversion.