PySpark - Apache Spark Programming for Beginners (2026)

Create a Databricks free edition account with a step-by-step signup, email verification, and login, then customize preferences like dark mode and editor theme.

Explore spark dataframes to process and transform data in memory, using dataframe APIs for select, filter, join, and aggregate, with hands-on practice loading sf-fire-calls.csv and saving results to a table.

Learn to read the sales sample csv with a defined schema on read, perform exploratory data analysis, and fix data type and schema issues in spark data frames.

Learn to add, remove, and rename columns in a Spark data frame using withColumn, withColumns, and case when expressions, and to define schema for efficient data frame creation.

Learn to write dataframe column expressions in Spark using select expr and withColumn, rename and compute arrival and departure dates, and compare SQL-style and column-based approaches.

Read the flight time data frame, filter for 2000-01-16 from US to AUD, compute delay as actual minus scheduled arrival, sort by delay, take three, and collect results into Python.

Transform unstructured data with Apache Spark using an AI prompt to extract log records into a structured JSON format with IP address, visit timestamp, visit resources, and referring URL.

Learn to work with numbers in Spark data frames by writing mathematical expressions, applying round and aggregate functions, creating total value columns, and performing analysis with describe, summary, and percentile.

Master string manipulation in Spark data frames using concat, concat_ws, and to_date to create dates, and format_number and format_string for presentation; explore translate and replace for data cleaning.

Explore complex data types in spark: struct, array, and map, parsing json fields with from_json and building optimized offline tables for fast analytics, including country-wise counts.

Explore five Spark join types beyond the basics—natural, cross, self, left semi, and left anti joins—using real examples with member, bookings, and facilities data.

Develop a local Spark application in Python using PyCharm by creating a Spark session, loading survey.csv into a data frame, and performing a group by country count.