Databricks Certified Associate Developer for Apache Spark 4

Understand big data as huge, fast, and complex data that traditional databases cannot handle, driven by the three v's—velocity, volume, and variety—and feeding modern platforms like Apache Spark.

Compare monolithic and distributed architectures, explaining vertical vs horizontal scaling and how adding machines enables distributed computing, with Apache Spark as the framework for scalable big data processing.

Explore the Hadoop distributed file system and map reduce, learn how data is split into 128 MB blocks across machines, enabling parallel processing for Spark architecture.

Explore how map reduce processes data across eight partitions using mappers to emit key-value pairs, shuffles data to reducers, and writes final results to HDFS, emphasizing data locality.

Apache Spark is a unified distributed computing engine that processes big data in memory, as an alternative to Hadoop MapReduce and up to 100x faster than MapReduce.

Explore Spark architecture by seeing how a cluster of interconnected nodes uses a cluster manager to allocate a driver and two workers, with an application master container guiding execution.

Explore the difference between Spark session and Spark context, and how Spark context, SQL context, and Hive context merged into Spark session, with Databricks hosting practical, hands-on Spark learning.

Learn to set up an Azure account, access Azure Databricks, and configure a Databricks workspace and compute, including pay as you go and Spark runtime basics.

Set up Unity Catalog, access the metastore, and configure the Databricks admin console to create and attach a metastore and external locations for data in a storage account.

Create a Databricks notebook, attach a cluster, and run PySpark code in cells. Databricks provides a default Spark session, and you can explore Spark UI and SparkContext vs SparkSession.

Learn to import notebooks and the .dbc Databricks file to your folder, using manual ipynb uploads or a single .dbc import that creates a complete project with notebooks and folders.

Explore the inner workings of driver and worker nodes, including the application master container, JVM main, PySpark driver, and Py4j translation, plus optional Python wrappers on workers.

Explore transformations in apache spark, distinguishing narrow transformations that operate per partition without shuffling from wide transformations that require data shuffling for aggregation or grouping.

Understand how spark lazily evaluates transformations to build an optimized plan. Execute only after an action triggers the run, with display or collect as examples.

Learn lazy evaluation in PySpark as you build a dataframe, apply transformations like select and filter, and execute only when you run an action such as display.

Explore how the Spark Catalyst Optimizer transforms code into logical and physical plans, uses explain to reveal scanning, filter, and project steps, and navigates the Spark UI and DAG.

Explore how the Spark ecosystem sits on resilient distributed datasets and uses data frames, data sets, SQL, and structured streaming to power MLlib and advanced analytics.

Shows why dataframe API with schema enables catalyst optimization, contrasts with rdd code readability, and clarifies partitions, execution plans, and driver vs executor fault tolerance.

Learn how Spark turns code into jobs, stages, and tasks via lazy evaluation, with actions triggering jobs that split into stages and tasks, organized by narrow and wide transformations.

Explore the spark-submit command as the tool to submit spark applications to a cluster, including master, deployment mode, executors, and memory while noting that Databricks manages infrastructure behind the scenes.

Explore the jobs, stages, and tasks flow in traditional Apache Spark, including repartition, select, filter, group by, and the shuffle read/write process, with insights on AQE and partitioned tasks.

Learn how repartition increases partitions to boost parallelism, at the cost of a shuffle, while coalesce merges partitions as a narrow transformation without shuffling, helping optimize partition counts.

Explore how Spark partitions are created and distributed, and learn how coalesce and repartition affect data layout. Use partition ID analysis to diagnose skewness and optimize distribution.

Master Spark query plans by exploring unresolved, logical, optimized, and physical plans, and learn how repartition, coalesce, grouping, and adaptive query execution affect execution via explain outputs.

Learn to read data in PySpark using spark.read with csv, json, and parquet formats, including header, infer schema, and multiline json, yielding scalable data frames.

Read data from a JDBC source in Apache Spark by creating a DataFrame with a JDBC URL and connection properties, using spark.read.jdbc or spark.read.format('jdbc') on orders.

Handle malformed records in Spark by choosing permissive, drop malformed, or fail fast modes, which store, drop, or fail on corrupted records in CSV and JSON inputs.

Define schema for raw data by enforcing a custom schema using struct type or DDL schema, illustrating when to replace inferred schema for CSV, JSON, and Parquet data.

Master the select transformation in spark to prune a dataframe by selecting city, customer ID, and country using the select API and column-name syntax.

Learn how to rename selected columns with alias in Spark dataframes, using the column method to assign meaningful aliases like customer city and customer country.

Learn the filter transformation in Spark dataframes to filter by order status, extract returned orders. Use is in for multi-value filtering and combine with or for readability and maintainability.

Rename a column in a dataframe in place with withColumnRenamed to change order status to order status info, highlighting in-place changes versus aliasing.

Unlock the withColumn API to add or modify columns in Spark dataframes, using literals with lit, regex replace with regexp_replace, and chaining transforms like calculating total price and rounding.

Type casting helps data engineers align data types for joins and aggregations, converting columns like order id from integer to string or vice versa using cast and the withColumn function.

Master sorting dataframes in PySpark with the sort API, using single-column and multi-column orders, including ascending and descending flags for date and quantity.

Apply the limit transformation to quickly fetch a subset of records for debugging and quick analysis, using df.csv.limit(n) or display, and relate it to pandas head and SQL top.

Drop removes unnecessary columns from a data frame as data moves to the transformed layer, creating a lean final data warehouse.

Learn to drop duplicates in Spark dataframes, applying full-record deduplication or deduplication on a subset of columns to prevent downstream errors.

Discover how to perform union with two data frames in Spark, then apply union by name to align columns by name when schemas are in different order.

Explore date functions in PySpark to handle current timestamps, add or subtract days, compute date differences, and format dates for consistent cross-source data.

Master PySpark string functions in Databricks by applying SQL-like transformations, such as upper and lower case, and measuring string length on dataframe columns like order status.

Learn how to split a shipping address column into street address and city using a comma delimiter, then apply indexing to extract values into new columns with column operations.

Explore the explode function in PySpark inside Databricks to expand array columns into rows, with a note on using explode outer to preserve null values.

Learn to filter arrays directly with the array contains transformation related to explode, using the array contains method to identify elements like city 1 without expanding to new columns.

Apply groupby to aggregate sales by product id, compute total price, and identify flagship products; extend grouping to include customer id for top customers with total and average sales.

Explore approximate count distinct, a group by aggregation that yields approximate distinct counts to save resources. Compare approximate and exact counts on product and customer IDs.

Explore the collect_list transformation in PySpark to group by customer and collect all ordered product IDs into a list per customer, enabling per-customer product analysis.

Discover pivoting in PySpark to analyze orders by customer and status, using groupby, pivot, and agg count to create a matrix of completed, cancelled, and shipped orders.

Learn how to implement if-else logic in PySpark using the when otherwise construct, build a return flag with conditional branches, handle nulls, and combine multiple conditions with and.

Explore mastering PySpark joins, including inner, left, right, full, and anti-join, with concept diagrams and practical df1, df2 examples.

Master window functions to perform row-based calculations in SQL and PySpark, covering row_number, rank, dense_rank, partition by, and order by, with deduplication and cumulative sum and moving average.

Learn to compute a cumulative sum with Spark window functions, turning row sums into per-row totals using over with order by year and unbounded preceding to current row.

Discover how to create and use PySpark user-defined functions (UDFs) to apply custom transformations on data frames, including registering functions with UDFs and the decorator approach.

Discover how to implement a user defined table function (udtf) as a class with eval to split a text column into a word-per-row data frame in spark.

Master calling user-defined functions with call udf in Spark, compare it to direct udf usage, and learn to register with Spark.udf.register for readability in dataframes and Spark SQL.

Learn how to use concat and concat_ws in PySpark to concatenate multiple columns with a separator, compare approaches, and produce clean, reusable code.

Learn how to write and store PySpark transformed data using four output modes—overwrite, append, error, and ignore—with Databricks Spark 4, across formats like CSV, JSON, Parquet, and Delta.

Explore writing data in multiple file formats, including CSV, JSON, Parquet, and Delta Lake, and learn how to convert data frames between formats and use append mode.

Explore how the Delta format (Delta Lake) adds acid properties and commit/rollback to parquet data via a Delta log of json files.

Apply upsert in PySpark with Delta Lake merge to update existing records and insert new ones into the Delta table, highlighting overwrite behavior and the Spark 4.0+ merge into API.

Learn to read and flatten complex json with PySpark, explode arrays, and fetch nested fields to turn messy json into clean tabular dataframes for analysis.

Master spark file source options, including ignore corrupt files, ignore missing files, path glob filter, and recursive file lookup for parquet, csv, and json in hierarchical folders.

Learn Spark SQL to work with data sources, perform complex joins and case when statements, create a dataframe, turn it into a temporary view, query Spark SQL, and display results.

Create a data frame from a temporary view using spark sql, then continue transformations in spark data frames, illustrating a hybrid approach that blends spark sql and spark data frames.

explore global temporary views and their cluster-scoped access across notebooks attached to the same cluster, and contrast them with session-scoped temporary views by creating and querying a global temp view.

Learn to use Spark SQL DDL commands to create and register permanent tables in the Unity Catalog, including schema creation, inserting data, and querying actual tables.

Apply joins in Spark SQL to combine two data frames using a left join, define a join condition on id, and explore creating a data frame from the join results.

Explore upsert in Spark SQL using merge into, covering when matched updates, when not matched inserts, and approaches with temporary views, delta objects, and Spark 4.0 merge into API.

Master partition by in Spark SQL, an optimization technique that stores data as partitions in a data lake to speed queries. Create Delta or Parquet tables and perform DDL commands.

Explore the Spark SQL explain command to view physical and logical query plans, including photon optimization and metastore paths when querying a Spark catalog table.

Use Spark SQL to write exact SQL code, including common table expressions, case statements, and modular arithmetic, to create inner queries and derive even or odd flags.

Learn Spark SQL scripting statements, including for loop, while loop, case statement, and if statement, with practical examples of summing odd numbers and conditional outputs.

Master spark sql auxiliary statements to inspect objects with describe, refresh, and show commands; explore databases, schemas, tables, and table properties, partitions, and delta metadata in unity catalog.

Master Spark SQL aggregate functions and advanced operations, including array aggregate and collect list and set. Learn to compute max, average, median, and mode with examples that translate to pySpark.

Explore struct and map in spark sql to create dictionary-like json structures and key-value pairs, using struct, named struct, and map functions to extract keys and values.

Master Spark SQL datetime functions by using current timestamp, add months, convert time zones, and work with Unix time and date parts such as year, month, and day.

explores spark sql window functions like lag and lead to fetch previous and next values over partition by order by, and ntile to bucket rows.

Explore Spark SQL array functions, from creating arrays and appending values to inserting at positions, checking containment, and compacting nulls, with practical PySpark examples.

Learn to configure Spark SQL with spark.conf.set or set, and review a concise revision sheet of key properties for in-memory columnar storage, file based source tuning, shuffle, join, and AQE.

Learn to create and register user-defined functions (UDFs) in Spark SQL and PySpark, using Python def, the create function command, and spark.udf.register, while preferring built-in functions whenever possible.

Learn how to read files directly with Spark SQL by creating temporary views for CSV and JSON data, then extend to Parquet, Delta, or ORC formats.

Query files directly with Spark SQL by creating temporary views on top of files. Or query a data frame directly with Spark SQL, without a temporary view.

Learn to query data directly on files with Spark SQL using the file format connector, supporting JSON, CSV, and Parquet, without creating a temporary view or data frame.

Explore how Spark performs joins under the hood, comparing shuffle sort-merge, shuffle hash, and broadcast joins, and learn to choose the best physical plan for inner joins.

Describe how Spark's shuffle sort merge join shuffles data by hashing IDs to 200 partitions, then sorts and merges within partitions for efficient joins.

Learn how shuffle hash join speeds joins by hashing the smaller partition in memory after shuffling, and when Spark prefers this over shuffle sort-merge joins.

Explore how Spark UI helps you understand joins between dataframes, including sort-merge join and hash join decisions. See how partitioning, exchange, and AQE reshape the plan and performance.

Learn how broadcast join in Spark broadcasts a small data frame to all executors to avoid shuffling the large data frame and speed joins between big and small tables.

Spark automatically uses a broadcast hash join for small dataframes (around 5 MB) with a larger dataframe; enforce it by using broadcast on the smaller dataframe.

Explain driver memory management, including JVM heap and overhead memory, with spark.driver.memory and spark.driver.memoryOverhead allocating 2 GB plus 10% (or 384 MB, whichever higher); show fetches only one partition.

Explain why a driver out of memory error happens when you run df.collect on a 10 GB dataset distributed across executors, with a 2 GB driver memory and partitions.

Learn how Spark executor memory divides heap into reserved, Spark pool, and user memory, with a 50/50 storage versus execution split and caching via storage memory.

Apache Spark unified memory balances execution and storage memory, uses LRU eviction, and moves the divider to optimize caching, memory management, and data processing.

Spark spills data to disk when executor memory runs out, evicting intermediate results to free space and continue processing.

learn why executor out of memory occurs during a group by on skewed data, when a large partition cannot fit in pool memory, forcing disk spills and possible failure.

Explore off heap memory and PySpark executor memory, including serialized data outside the JVM heap to reduce GC overhead and enable caching and Python–JVM serialization for pandas UDFs.

Explain edge node and deployment mode in spark, showing how an edge node guards cluster manager access and enables code submission through a secure gateway for junior developers.

Explore deployment modes in spark, including cluster mode and client mode, where a driver resides on the cluster or edge node, and note local mode for learning.

Discover adaptive query execution (AQE) in Spark 3.0, a runtime optimization layer that dynamically coalesces partitions, handles skewness, and refines join strategies to improve performance.

Discover how AQE coalesces partitions to reduce tasks and balance workload, turning many small partitions into fewer, efficient groups during Spark data processing.

AQE splits large skewed partitions into smaller ones to boost Spark performance, coalescing small partitions and applying the five-times median rule with a 256 MB threshold by default.

Observe how AQE optimizes join operations at runtime by using actual data sizes to switch from sort merge join to broadcast join, saving resources when pruning reduces one side.

Learn how to cache and persist dataframes in Apache Spark, using storage and execution memory to avoid recomputation and improve performance through DAG optimization.

Explore persist and cache concepts in Spark, focusing on memory and disk storage levels for dataframes and RDDs. Learn eviction, recompute, and spill strategies.

Explore storage levels for persisting dataframes, including memory only, memory and disk, disk only, memory only two, and off heap; learn to cache with df.cache and understand rdd implications.

Learn to cache a data frame in PySpark, inspect storage in the Spark UI, and understand storage levels like memory and disk to avoid recomputation.

Persist PySpark dataframes using storage level disk only. Import storage level, apply tf.csv.persist, and inspect the Spark UI to see disk-only storage.

Learn how to clear cached dataframes with unpersist in Spark, replacing the misconception of uncache; remove dfcsv and dfjson data to free memory during ad-hoc analysis, verified in Spark UI.