Apache Spark 3 Programming | Databricks Certification Python

Learn to sign up for the free Databricks Community Edition, verify your account, create a cluster and notebook, and run Python code in the notebook using shift+enter or ctrl+enter.

Access course materials, data, and notebooks; upload data and code to the community edition cluster; verify uploads via dfs and workspace, and manage a single active cluster.

Master the Apache Spark architecture, including the driver, executors, spark context, and cluster manager, and distinguish cluster mode, client mode, and local mode execution with spark submit.

Explore how a dataframe represents a distributed, immutable table of rows and columns, with a defined schema and lineage of transformations that trigger actions in Spark.

Explore how to create and work with a data frame in PySpark in Databricks notebooks, using create data frame API, display results, and count rows with column names and schema.

Learn to rename a data frame column and change data types using cast, then select and cast columns to achieve the desired schema.

Explore selecting columns from the earthquake dataset using Park Dot and Sequel's style API, rename and cast columns, and drop unused fields for efficient downstream processing.

Learn how to perform filtering operations in Spark on earthquake data, keeping entries with magnitude greater than 6.0. Cast columns, apply filters, and count results for insights.

Learn to drop columns in a Spark dataframe using the drop API, by name or multiple columns, and remove rows with nulls using any or all conditions.

Learn union of dataframes by rows, preserving columns while increasing rows, and apply distinct to keep unique rows. Compare union with join using example science and English scores.

Sort a dataframe by subject_name and subject_score in ascending or descending order using three methods: sort, order, and a sql-like approach with a temporary view.

Group and aggregate data using group by, min, and mean to find each student's lowest score across two subjects, then rank students by total score in descending order.

Learn advanced aggregation in Apache Spark with Python to compute average and variance of CO2 levels across 2016–2018, using dataframe, select, and collect.

Explore how inner joins merge data frames on keys, contrast with unions, and explore various join types such as left joins, using student score examples to show column expansion.

Explore partitions in Spark data frames, and learn how coalesce and repartition affect partition counts, shuffling, and overall performance in a cluster.

Learn how accumulators and broadcast variables enable scalable Spark processing by aggregating data across executors and caching lookups on all nodes.

Explore how persist and cache optimize dataframes by storing partitions in memory or disk with serialization, enabling reuse across actions.

Explore time and date functions in PySpark, extract timestamps, compute day of week and date differences, filter data between two dates, and count earthquake events in a sample dataset.

Explore reading and writing different file formats in Spark by defining schemas, handling data types and nullability, and using dataframe read and write with CSV and Parquet examples.

Navigate the spark code lifecycle from client request to completion, detailing resource allocation by cluster and node managers, driver and executors, and parallel task execution across the cluster.

Explore the Spark code lifecycle from inside the framework: convert user code to logical and physical plans, optimize with the Catalyst optimizer, and execute with staged tasks and partitioned shuffles.

Learn about the adaptive query engine in Spark and how AQE optimizes execution at runtime by using statistics to adjust partitions, join strategies, and SKU joins for better performance.

Explore creating and registering user defined functions in spark, applying them to data to increment values, and recognizing the performance cost of external UDFs not optimized by the framework.

Learn optimization techniques to speed up Spark code, including in-memory caching, efficient partitioning with repartition or coalesce to reduce skew, and selecting suitable file formats such as Avro and Parquet.

Discover the Databricks Spark 3.0 exam layout, with online proctoring, 60 questions, 120 minutes, and a 70 percent passing score, plus coverage of Spark architecture and API applications.

Explore how linear regression establishes a linear relationship between independent and dependent variables using slope and intercept, analyzes mileage data, and outlines data read, preprocessing, fitting, and evaluation steps.

Perform a hands-on regression with Weisbach library. Define schema, handle missing values, select features, assemble vectors, split data into training and testing, train linear regression, and compare with boosting methods.

Examine classification as a supervised learning task with discrete targets, using the iris dataset of three species and four attributes to predict species.

Learn to build a classification model in Spark by preparing balanced data, encoding labels, creating feature vectors, and evaluating logistic regression and a depth-limited decision tree to avoid overfitting.