Data Analytics with Pyspark

Explore what big data means, including the four V's—volume, velocity, variety, and veracity—and gain an overview of Spark concepts, cluster deployment, and PySpark fundamentals.

Master the resilient distributed dataset (RDD) as Spark's core for parallel, fault-tolerant data across a cluster, driven by transformations, actions, and lazy computation.

Explore RDD operations using lazy transformations like flatMap, map, and filter, and actions like reduceByKey and collect to count words from a text file.

Explore pair RDDs in pyspark, focusing on key-value data, hash partitioning, and core transformations such as map, flatMapValues, reduceByKey, groupByKey, aggregateByKey, and combineByKey, plus lookups and joins by key.

Create PySpark dataframes from lists, RDDs, and external sources (files, databases) and define schemas with struct types; print schema and handle empty dataframes.

Learn how the PySpark column class provides operators and functions to manipulate data frame columns, access and alias columns, filter data, concatenate names, cast types, and apply when expressions.

Explore sql aggregate functions in pyspark, including count distinct, avg, collect_list, collect_set, max, min, mean, skewness, standard deviation, sum, and variance on a salary dataset.

Explore window functions in Spark SQL to perform ranking, analytic, and aggregate operations within partitioned and ordered data windows.

Learn to create and customize matplotlib plots within PySpark workflows, including selecting columns, plotting salaries and medians, labeling axes, styling lines, and adjusting layout for clear visualizations.