Advanced Data Analysis & Wrangling with Python Pandas

Install Python and pandas via python.org for Windows or Mac. Use pip to install or update pandas and numpy, and add 3 on Mac for Python 3.

Install jupyter notebook or jupyter lab with pip, then launch a local development server and use your browser as the interactive Python IDE.

Explore the relationship between numpy and pandas, with numpy as the numerical backbone and pandas as a higher level data wrangling tool built on numpy.

pandas basics center on series, the foundation of data frames, created as a one-dimensional array from lists or numpy. Learn about index behavior, including string-based indexing, slicing, and common gotchas.

Name a pandas series, view its max and min, and change the index. Convert to numpy with to_numpy, avoid the deprecated values attribute, and use np.nan for missing values.

Explore data frames in pandas by creating data frames from series, dictionaries, and numpy arrays, assign meaningful column names and indexes, and understand the distinction between indexes and columns.

Explore fundamental DataFrame operations and common gotchas in pandas, including how square-bracket indexing targets column names, using dot loc/iloc for rows, converting to numpy with to_numpy, and updating the index.

Access and compute with a data frame by treating columns as dictionary keys, create new columns like population_2010, apply growth rate calculations, and rename columns to avoid method-name clashes.

Explore useful pandas data frame methods for quick inspection and analysis, including head, tail, sample, columns, describe, info, and shape, plus memory usage and data types considerations.

Learn to add and drop columns in pandas dataframes, create derived columns from existing ones, and remove rows or columns using drop and del with in-place options.

Learn how to read and write data with Python pandas, covering csv, parquet, and json formats. Explore csv limitations, advantages of parquet, and practical io strategies for real data analysis.

Learn how to read a CSV dataset with pandas in a Jupyter notebook, inspect structure with head and info, and recognize missing values and data types like strings and floats.

Read a csv with date time columns by skipping top header and footer, using pandas read_csv with skiprows and skipfooter, and converting numeric dates to datetime via a string-to-datetime step.

Learn how to save a pandas data frame from the Fama-French dataset to CSV without the index, using index=False, to keep files clean and compatible with tools like R.

Learn how to save a data frame from pandas to a csv file using to_csv, and why always set index=False to avoid exporting the index as a column.

Learn how to read and write parquet files with pandas, install pyarrow and fastparquet, and compare parquet to csv in terms of space efficiency, speed, and type preservation.

Learn to read and write tab-delimited files with pandas by saving with a tab separator, removing the index, and reading with read_csv using sep='\t', including date parsing.

Learn to fetch json from the web using requests, convert the payload to a pandas data frame with json_normalize, and save or read json with to_json.

Learn how to select and filter data in pandas data frames, assign new data, and navigate common indexing gotchas using a California restaurant Yelp dataset.

Identify common pandas gotchas in slicing data: distinguish row slicing by index from column selection by names, and learn two data accessors to slice and select data more reliably.

Explore the pandas .loc selector to slice data by row indices and column names, with inclusive start and end, multiple syntax options, and common gotchas.

Learn to conditionally modify rows with the .loc selector, duplicating data to avoid altering the original, and apply prefix changes to selected names.

Explore the pandas iloc and loc selectors, highlighting integer-based indexing, end behavior (iloc excludes end; loc includes end), and guidance on when to use or avoid iloc with non-integer indexes.

Reset the index to default order using reset_index with job=true, convert the index into a normal column when needed, and save the result back into data.

Learn to filter pandas data frames with boolean masks, using city and rating criteria via .loc and logical operators to perform multi-criteria selections.

Learn to chain multiple pandas operations to filter restaurants by city Pomona, price range cheap, and ratings above threshold, with stepwise builds and final selection.

Sort a dataset by a single column using pandas sort_values, returning a new dataset ordered ascending and placing missing values at the bottom.

Use pandas sort_values to sort a data frame by multiple columns in order: city, price range, then rating; control with ascending flags and in-place or out-of-place saves.

Learn to count rows and columns in a data frame using shape, index, and len, and compare with the count function for non-missing values and NaN excluded.

Count unique values across columns with n unique, compare unique counts and value counts, and identify duplicates to guide data cleaning before analysis.

Learn to clean duplicates in pandas by using the duplicated method with subset on name, street address, and city, and choose keep first, last, or all to produce unique data.

Drop duplicates with pandas using subset columns like city and street address, and choose to keep first, last, or none. Learn how this affects shape and facilitates quick data analysis.

Learn to detect and handle missing data in pandas with isna and notna, using stock A, B, and C datasets and a date-indexed dataframe for clean filtering.

Explore how missing values propagate in pandas computations. Learn how to handle them with explicit conversions or dropna, including subset and axis strategies.

Explore alternatives to dropping data by filling missing values in a dataframe, using a scalar like zero and showing how to apply it to the data.

Learn how to fill missing values in time-series data using forward fill and backfill, with limits and chaining. Decide when these methods are appropriate, unlike non-time-series data.

Learn to fill missing values in pandas dataframes with the column mean, using df.mean and the where function, and understand axis nuances for row vs column means.

Use the replace method to substitute Na values with zero and infinities with NaN, preparing data for clean storage in SQL databases.

Learn how to detect, drop, replace, and interpolate missing values in time-series data using pandas, including forward fill, backfill, and straight-line interpolation with df.interpolate.

Explore data aggregation and transformation in Python Pandas, learning how to summarize columns with functions like sum, mean, and max, while transformation preserves shape by applying values element by element.

Learn aggregation basics in pandas by computing mean, median, max, min, and standard deviation for real estate price, using multiple functions on numeric columns from a Realtor dataset.

Count on pandas to compute multiple statistics for multiple columns at once, producing a two-dimensional table with metrics like mean, median, maximum, and standard deviation across bedrooms, bathrooms, and price.

Apply different aggregate functions to specific pandas columns with a dictionary mapping columns to functions. Compute per-column statistics like max for bedrooms, median for bathrooms, and mean for acres.

Use idxmax and idxmin in pandas to locate the index of the maximum and minimum prices, then slice the data to inspect the corresponding rows.

Explore pandas built-in aggregation functions, including count and size, and compare non-missing values to all values. Learn to locate missing prices in the price column with data.shape and data.loc.

Explore Pandas statistic functions by computing variance, standard deviation, standard error of the mean, skewness, sum, and product on a sample dataset, and compare these measures to understand data distribution.

Learn how to use user defined functions (UDF) for aggregation in pandas, apply multiple functions across columns using dictionaries, and define custom aggregations like mean minus median and range.

Explore transformation in pandas by ranking prices and computing one day changes in forex data, preserving the original data dimensions and creating new change columns.

Explore time series transforms in pandas, including lag and shift, compute diff and pct_change, and build cumulative sum or product while handling data views and future data cautions.

Explore applying multiple transformation functions with pandas transform across several columns, including shift and diff, using axis settings and dictionary mapping, while avoiding aggregation functions.

Define and apply user defined transformation functions in pandas using transform, ensuring the output matches input dimensions. Explore numpy broadcasting, parameter handling, and standardizing data across columns.