PySpark Crash Course - Learn Analytics with Spark, Quickly!

Install PySpark, create a Spark session as the access point to Spark functionality, then upload and read a CSV into a data frame, using header inference to set column names.

Explore and understand a loaded data frame using Spark methods like show, print schema, describe, and count, then select columns to create a new data frame for subsequent analysis.

Create a custom schema in PySpark with a struct type to set bytes fields as integers and transaction time as float, then read csv with header using this schema.

Learn to handle null values in PySpark by applying field-specific fill rules, dropna, and mean or median imputation, with hands-on examples of using df.fillna, dropna, and agg.

Create a temporary view named df sql from the dataframe and run Spark SQL to select columns, filter by country equals England, and perform aggregations with group by.

Group a Spark data frame by country to compute sum of deal bytes and average of ul bytes with agg and aliasing. Extend by grouping on platform and counting rows.

Create calculated fields in PySpark by adding total bytes from dl_bytes and ul_bytes, extracting year, converting seconds to milliseconds, and classifying device type with a when statement.

Learn to handle duplicates that can skew analysis by dropping duplicates globally or by user id, using groupby and count to identify them, then overwrite the dataframe.

Write a dataframe to files with Spark using df.write.csv, and use coalesce to produce a single file, while exploring JSON, Parquet, and ORC outputs.

Explore how to use when statements in PySpark to tag a dataframe column by value and add a case example column marking transactions as large or small.

Learn to use case statements in PySpark to fill null values by testing is null. Create a new column using with column and daisy-chain functions for conditional logic.

Learn to implement case statements in PySpark by adding a transaction size column to a dataframe, using when with and/or conditions on dl_bytes and ul_bytes, and review the results.

Explore how to build a multi-branch PySpark case statement to classify a data frame column 'transaction size' into large, medium, small, or tiny via top-to-bottom evaluation and backslash line continuation.

Practice crafting a PySpark case statement to classify transactions: return active when transaction time is greater than 3600, otherwise return inactive.

Learn to solve the problem by adding a solutions column to a PySpark data frame that marks rows as active when transaction time exceeds 3600 (later 1.6) and inactive.

Explore PySpark window functions to rank transactions within each country by descending dl_bytes, partitioning by country, ordering by dl_bytes, and applying the rank function.

Learn how to use the row_number window function in PySpark to assign chronological row numbers within partitions, such as by country, ordering by date time.

Explore lag and lead window functions in PySpark, comparing previous and next rows by a time-ordered window, handling nulls at dataset ends.

Explore the sum window function to compute category totals and per-transaction percentages within a user partition, returning a sum column for chairs, tables, and overall sales.

Compute per-domain averages of DL bytes consumed using a window function, and print the same average transaction size for each domain row, preparing to compare domain-wide results.

Compute the per-domain average bytes using a window partitioned by domain, then add a new column average bytes by applying the DL bytes over that window.

Filter a dataframe to match rows where country equals Wales (case sensitive) and domain equals hopkins.org, using and conditions; or use or conditions so Wales or hopkins.org qualifies, then show.

Demonstrate using not conditions to filter PySpark dataframes by domain and country, selecting domain hopkins.org with country not Wales and excluding both criteria to show broader results.

Learn to filter a dataframe in PySpark by checking if the country column is in a Python list of countries, enabling consistent, reusable filtering across the code.

Filter dataframes in PySpark by substring matching to handle inconsistent country spellings, using like and contains to select rows where the country column includes a target pattern.

Create a PySpark udf to sum download and upload bytes, then apply it to a data frame. Define the function, register the udf, and specify the integer return type.

Define a string-returning UDF in PySpark to convert the http method field to lowercase and create a new processed http column using withColumn.

Learn how to define and register a UDF in PySpark to convert a transaction time column from seconds to milliseconds by multiplying by 1000 and applying it to a dataframe.

Create a UDF in PySpark to multiply the transaction time by 1000, add a new milliseconds column, and apply rounding to zero decimals for consistent results.

Learn to extract year, month, and day from a PySpark date time column, chain date-time transformations with dot with column, and reformat dates using the date format function.

Import the day of week function, extract the day from a date time column, then classify 1 or 7 as weekends and 2–6 as weekdays with a case statement.

Compute a weekend or weekday label from the date_time field in PySpark by deriving a day_number from day_of_week, then using when and between to classify.

Master the basics of PySpark and explore advanced techniques like window functions, user defined functions, and DateTime functions, then apply your skills to real-world data.