PySpark - Apache Spark Programming for Beginners (2026)

Understand Apache Spark as a multi-language engine and framework for data engineering, data science, and machine learning on single-node or cluster setups. Learn its SQL, batch, and stream processing capabilities.

Examine the Spark system architecture, covering Spark core and RDDs, language wrappers, SQL and data frame APIs, structured streaming, ML lib, and Spark Connect, plus cloud deployment and storage.

Explain the Spark platform and development environments, detailing Spark core, libraries, resource manager, and storage. Compare cloud platforms like Databricks and EMR with on-premise Cloudera, and notebook versus IDE development.

Understand Databricks Cloud, a unified data intelligence platform built on Spark, MLflow, and Delta Lake, enabling on-demand and serverless Spark workloads on AWS, Azure, and Google Cloud.

Create a Databricks free edition account with a step-by-step signup, email verification, and login, then customize preferences like dark mode and editor theme.

Set up your Databricks free environment, import the spark programming dbc, and initialize a dev catalog, spark_db database, and data sets volume to load diamonds data from GitHub.

Explore spark dataframes to process and transform data in memory, using dataframe APIs for select, filter, join, and aggregate, with hands-on practice loading sf-fire-calls.csv and saving results to a table.

Learn spark data frames by reading data, applying transformations, and triggering actions to execute and display results, while understanding four categories: transformations, actions, data frame writer methods, and auxiliary methods.

Explore Spark DataFrame concepts, including the optimized query plan and explain plans. Understand immutable dataframes and composable transformations that return new dataframes.

Explore Spark data frame transformations and the data frame API through hands-on practice with select, where, distinct, group by, order by, and limit, using the SF Fire Calls table.

Explore five approaches to creating a spark data frame, including using a connector, reading from spark tables, sql, Python lists, and ranges.

Explore Spark data types, from numeric primitives (byte, short, int, long, float, double) to complex types (array, map, struct) and interval types, with Python and Spark SQL keywords.

apply schema on read in spark by defining a flight schema for a json file, ensuring correct field order and date types, and writing the data to a table.

Explore data type correction in spark and schema on read. Convert time fields into interval data types to enable straightforward sql operations and analysis on flight time data.

Learn to read the sales sample csv with a defined schema on read, perform exploratory data analysis, and fix data type and schema issues in spark data frames.

Learn to add, remove, and rename columns in a Spark data frame using withColumn, withColumns, and case when expressions, and to define schema for efficient data frame creation.

Learn to write dataframe column expressions in Spark using select expr and withColumn, rename and compute arrival and departure dates, and compare SQL-style and column-based approaches.

Learn row-level transformations in PySpark to filter records and remove duplicates, using four expression approaches, and apply distinct or drop duplicates on chosen columns.

Read the flight time data frame, filter for 2000-01-16 from US to AUD, compute delay as actual minus scheduled arrival, sort by delay, take three, and collect results into Python.

Transform unstructured Apache logs with Spark by extracting ip address, visit timestamp, visit resource, and referring url using regex, then refine timestamps and root url, comparing ai-based extraction.

Transform unstructured data with Apache Spark using an AI prompt to extract log records into a structured JSON format with IP address, visit timestamp, visit resources, and referring URL.

Explore how to work with nulls in Apache Spark, including equality checks with is null, null handling in logical and mathematical operations, and using nvl for safe aggregations.

Learn to work with numbers in Spark data frames by writing mathematical expressions, applying round and aggregate functions, creating total value columns, and performing analysis with describe, summary, and percentile.

Master string manipulation in Spark data frames using concat, concat_ws, and to_date to create dates, and format_number and format_string for presentation; explore translate and replace for data cleaning.

Explore date types in Apache Spark, using date functions to convert strings to dates, perform date arithmetic, and format results while handling invalid dates as null.

Discover how to process timestamps in Spark by converting strings to timestamp with formats, using two_timestamp and try_timestamp, and understanding timestamp components and default utc time zone.

Explore handling time zone information in spark by contrasting timestamp with time zone and timestamp without time zone, using session time zone and convert_time_zone to normalize to utc.

Explore complex data types in spark: struct, array, and map, parsing json fields with from_json and building optimized offline tables for fast analytics, including country-wise counts.

Load json data with spark using the json connector and a schema, then query complex types to analyze country counts and spark experience.

Learn how to use the variant data type in PySpark to store semi-structured fields from a csv, parse json without a fixed schema, and query with variant explode.

Introduce joins in Apache Spark and explore eight join types, including inner, outer, natural, cross, self, semi, and anti joins, with hands-on data prep using facilities, members, and bookings tables.

Master inner joins with Spark dataframes to combine members and bookings, alias tables, filter Smith with more than five slots, and sort by first name and slots.

Explore outer joins in Apache Spark, including left, right, and full outer joins on DataFrame APIs, with Darren Smith scenarios and nulls first sorting for bookings.

Master lateral joins in PySpark data frame APIs to query a right table for each left row, enabling per-parent top N results and table-valued function calls.

Explore five Spark join types beyond the basics—natural, cross, self, left semi, and left anti joins—using real examples with member, bookings, and facilities data.

Discover simple aggregations in Spark data frames using sum and average to compute total earnings and average booking value, and learn the agg transformation for broader groupings.

Discover how to create scalar Python UDFs, Pandas vectorized UDFs, and UDTFs in Apache Spark, and register them for Spark SQL to use in dataframes.

Explore vectorized Pandas UDFs in Spark, learn how they process input as Pandas Series or Pandas DataFrame in blocks, use Arrow serialization, and boost performance over Python UDFs.

Develop a local Spark application in Python using PyCharm by creating a Spark session, loading survey.csv into a data frame, and performing a group by country count.