Databricks Certified Associate Developer - Apache Spark

Sign up for an Azure account via portal.azure.com, log in, and set up the Databricks environment for the Databricks Certified Associate Developer for Apache Spark 3.0.

Learn to set up an Azure-based Databricks workspace, sign in to portal.azure.com, create a resource group and Databricks environment, and access the Databricks console for Spark 3.0 certification practice.

Master the prerequisites for the Databricks spark developer certification by understanding spark architecture, adaptive query execution, and the spark dataframe API for tasks like selecting, filtering, joining, and writing data.

Set up a Databricks single-node cluster on Azure for the Databricks Certified Associate Developer exam, using 9.1 LTS, standard runtime, and minimal configuration to practice Spark 3 APIs.

Create and run a Databricks notebook linked to a running cluster, choose Python or Scala, execute hello world, verify Spark availability, and troubleshoot if needed.

Import the Databricks course material as a zip or DBC file into your workspace, then open the Databricks Certified Apache Spark Developer folders and notebooks to practice aligned LMS modules.

Explore Databricks notebooks and clusters to practice the course material, set up datasets, and run cells, syncing LMS modules with course folders.

Install the databricks cli with python and pip, configure it with a host url and token for azure or aws, verify by listing dbfs files and managing clusters and jobs.

Discover how to use the Databricks CLI to interact with DBFS, listing, copying, creating directories, and deleting files, with practical scripting for organizing datasets like yelp.

Set up retail datasets on Databricks using the CLI, installing and configuring the CLI, cloning repos, and copying data to DBFS.

Validate retail datasets in Databricks notebooks by listing files with percentage fs ls and using dbutils fs to verify paths for orders and retail_db_json.

Learn to create spark dataframes from Python collections and pandas dataframes, convert them into spark dataframes, and explore their use in a Databricks environment on Azure.

Create a single-column spark dataframe from a Python list using spark.create dataframe with a specified schema, and compare approaches with pandas dataframes.

Create multi-column spark dataframes from Python lists of tuples, learn to build single or multi-column frames, and optionally specify schema with column names using spark.createDataFrame.

Explore the pyspark row class and how to create row objects using varying positional and keyword arguments, inspect with collect, and access values by attribute names within pyspark dataframe workflows.

Convert a list of lists into a Spark dataframe by turning them into rows with PySpark Row, using list comprehension and star expansion, and optionally specify a schema.

Learn to convert a list of tuples into a Spark dataframe using Row, turning tuples into rows, and optionally apply a custom schema such as user_id int and user_first_name string.

Demonstrate converting a list of dicts into a spark dataframe via row, using varying arguments or varying keyword arguments, with schema control and deprecation notes.

Create a spark dataframe from a list of dicts to explore basic datatypes like bigint, string, boolean, double, date, and timestamp, and learn schema inspection.

Learn to specify a dataframe schema as a string when creating a Spark dataframe from a Python list of tuples, mapping int, string, boolean, float, date, and timestamp types.

Create a spark dataframe by passing a list of column names as schema, with data inferred types, and build data from a list of tuples using spark session.

Learn to specify a spark dataframe schema with pyspark.sql.types using struct fields and a struct type, choosing integer, string, boolean, date, timestamp, and float types, then create the dataframe.

Convert a pandas dataframe to a spark dataframe to handle missing attributes via NaN in a list of dicts, then inspect the schema with print schema after spark.create dataframe.

Explore how Spark creates dataframes with primitive and special data types, including array, struct, and map, and learn how Python lists and dicts convert to these types using Spark APIs.

Learn to handle array type columns in Spark dataframes by creating a users dataset, exploding arrays into records with explode and explode outer, and inspecting maps and structs.

Explore map type columns in Spark dataframes by converting Python dicts to maps, accessing values by key (mobile, home), and using explode and explode_outer, with column renaming for clarity.

Explore struct type columns in Spark dataframes, using PySpark Row to define predefined phone numbers with mobile and home, and access nested fields with dot and bracket notation.

Learn to select and rename columns on dataframes using select with column and with column rename, while preparing for column manipulation with UDFs and Spark SQL functions.

Create a Spark dataframe in a notebook and learn to select and rename columns, handling struct and array types like phone numbers and courses.

Explore narrow and wide transformations in Spark data frames, distinguishing shuffle-driven wide operations from row-wise narrow operations like select and filter, with topics on join, union, distinct, and group by.

Explore selecting columns in a spark dataframe using select with strings, lists, aliases, and expressions; project columns, derive full name with concat and col, and preview with show.

Explore Spark dataframe select expression to project data with SQL style syntax. Compare select and select expression, alias and concatenate, and use temp views.

Learn how to refer to Spark dataframe columns by dataframe names using select, strings, or the col function, and apply aliases for joins and temporary views.

Understand the spark col function and column objects: create from dataframe notation or col, select with strings or lists, and use cast, alias, date_format, asc/desc, contains.

Invoke functions via Spark column objects to create derived columns, such as building full_name with concat and lit and alias, and convert date fields to numeric with date_format and cast.

Learn how the lit function in Spark converts literals to column types to perform arithmetic on dataframe columns. See why using lit avoids errors when adding 25 to amount_paid.

Explore methods to rename Spark dataframe columns or expressions using alias, withColumn, withColumnRenamed, and derived columns, including when to apply withColumn versus select.

Learn how to name derived columns with withColumn in Spark, using full_name via concat and lit, and compute course_count from arrays with size.

Learn to rename dataframe columns with withColumnRenamed, changing id to user id, first name to user first name, and last name to user last name, while preserving column order.

Rename spark dataframe columns using alias and create derived fields, such as user_full_name by concatenating first and last names with a comma and space, using col, concat, and lit.

Learn to rename and reorder spark dataframe columns in one shot by selecting required_columns, then applying toDF with target_column_names and star to unpack the list.

Explore manipulating columns in spark dataframes, using pyspark.sql.functions, with emphasis on predefined functions, column helpers like col and lit, string and date-time operations, json processing, and practical examples.

Explore predefined spark dataframe functions in pyspark.sql.functions to manipulate data with select, filter, group by, and sort, using column names or expressions. Use date_format for derived values.

Create a dummy spark dataframe from a list of tuples with a single column named dummy to explore spark functions, current_date, and print schema.

Explore categories of functions to manipulate columns in spark dataframes, including string, date, and aggregate functions, plus arrays, maps, and structs, with practical usage examples.

Explore how to get and interpret help for spark functions like date_format, col, lit, concat, and concat_ws, and apply examples to understand arguments and return types.

Explore pyspark's special functions col and lit, converting string column names to column objects, using literals, and applying operations like upper, alias, and concat on dataframes.

Learn common string manipulation in spark dataframes, including concat and concat_ws for combining fields, and case conversions with upper, lower, initcap, and length to derive full_name and full address.

Learn to extract substrings from spark dataframe columns using pyspark.sql.functions.substring, applying fixed-length and variable-length records, including ssn and phone numbers, and cast results to integers.

Extract strings from Spark dataframe columns with split, access array elements, and explode into rows for analyses; learn through examples like addressing city and state and splitting phone numbers.

Explore how to pad strings in Spark dataframe columns with lpad and rpad. Create fixed-length records by padding numeric and non-numeric fields and concatenating them.

Explore trimming characters from spark dataframe string columns using ltrim, rtrim, and trim, with spark sql expressions for left, right, and full trimming, including non-space characters.

Explore date and time manipulation in Spark dataframes, using current_date and current_timestamp, and convert non-standard date or timestamp strings to standard formats with to_date and to_timestamp.

Apply date and time arithmetic in spark dataframes using date_add, date_sub, datediff, add_months, and months_between. Create a date_times_df, perform adds/subtracts, and compare with current_date and current_timestamp.

Explore how to use trunc and date_trunc functions on spark dataframes to compute week-to-date, month-to-date, and year-to-date reports, including format options and argument differences.

Explore extracting date and time components from Spark dataframes using pyspark.sql.functions—year, month, weekofyear, dayofweek, hour, minute, second—and apply them to current_date and current_timestamp columns.

Convert non-standard dates and timestamps to standard formats in Spark dataframes using to_date and to_timestamp, by selecting the correct input format for each pattern.

Explore how to use Spark's date_format to convert dates and timestamps to target formats, extract year, month, day, and Julian day, and cast results to integers in Spark dataframes.

learn how to work with unix timestamps in spark dataframes by converting between unix time and standard dates or timestamps, using unix_timestamp and from_unix_time, with casting and format options.

Explore dealing with nulls in spark dataframes using coalesce and nvl, and manage bulk nulls with dataframe.na functions like fill, replace, drop, and handling empty strings.

Use case and when in spark dataframes to apply conditional transformations with sql style expressions or api, and handle nulls with coalesce while categorizing ages from newborn to adult.

Explore filtering spark dataframes with filter and where on a users_df dataset, inspecting id, first name, last name, email, city, and nested phone numbers and courses.

Create a Spark dataframe from a list of user records, define phone numbers as a struct and courses as an array, then explore schema, columns, and dtypes for filtering.

Explore how to filter Spark dataframe using filter or where, employing SQL style and non-sql style conditions, including col-based and dataframe notation, to produce a new filtered dataframe and view.

Explore filtering Spark dataframes with filter or where, building conditions using equal, not equal, greater than, less than, between, contains, and in, including SQL style syntax.

Filter Spark dataframes using equal conditions on boolean, string, and numeric columns (such as is_customer, current_city, amount_paid) with non-sql and sql-style syntax, including handling NaN with is_nan.

Master not equal filtering in spark dataframes using non-sql and sql style syntax, and handle nulls and empty strings with is null checks.

Learn to filter spark data frames with the between operator, applying range conditions on last_updated_timestamp and amount_paid, using non-sql and sql-style syntax.

Learn to filter spark dataframes by handling null values using is null and is not null, with non-sql and sql style syntax, and distinguish null from empty strings.

Explore boolean operations for filtering Spark dataframes, including or, and negation, with true/false outcomes and practical examples for combining multiple conditions.

Explore boolean or on the same column in a Spark dataframe, replace it with in for multiple values, and learn null handling with is null and isin.

Master filtering in Spark dataframes using greater than, less than, and their inclusive forms with non-sql and sql style syntax, including date comparisons and null handling.

Learn to filter spark dataframes using a boolean and condition to select male customers with is_customer true, and compare sql and non-sql syntax for between date ranges.

Apply boolean or across Spark DataFrame columns to filter users who are not customers or have an empty city, using non-sql col syntax and sql style syntax.

Practice dropping columns from a dataframe using the drop function in Spark on Databricks, and explore common use cases to prepare for the certification.

Create and run a notebook to quickly build a Spark dataframe, then explore the drop function by removing columns and previewing five records with show.

Explore how the Spark dataframe drop function removes unwanted columns, with examples using column names or column objects, including excluding last_updated_ts and creating new dataframes.

Drop a single column from a Spark dataframe by using a string or column object, preview the schema, and verify the new dataframe, while noting that non-existent columns are ignored.

Drop multiple columns from a Spark dataframe by passing column names as strings to the drop function, preview with show; missing columns are ignored and all names must be strings.

Create a dataframe and define a pii_columns list containing confidential fields, then convert the list to varying arguments and pass them to drop to remove those columns.

Drop duplicate records in Spark dataframes using distinct, drop_duplicates, or drop duplicates; specify a subset of columns to drop duplicates based on key fields.

Learn how to drop null-based records from Spark dataframes using df.na.drop and dataframe.dropna, with thresh, subset, and how parameters (any or all), including scenarios for all-null or partially-null rows.

Explore sorting data in spark dataframe using core sorting concepts and APIs, practice with examples, and prepare for exam questions by gaining confidence in sorting operations.

Sort data in a spark dataframe by using a reusable notebook to create the dataframe and explore sorting examples, validating the notebook runs before reuse.

Explore sorting a Spark dataframe with ascending and descending orders, handling nulls, and composite and prioritized sorting; learn sort and order by APIs, and practical examples.

Sort a Spark dataframe in ascending order by any column, using string or column objects, including first_name, customer_from, and size of courses, with sort or order by and null handling.

Sort a Spark dataframe in descending order by a given column using string names or column objects, using desc, col, and related sorting functions in PySpark.

Learn how to handle null values while sorting a Spark dataframe using asc, desc, and nulls first/last variants on a selected column.

Explore composite sorting in a spark dataframe by sorting first by suitable for and then by enrollment, using sort or order by with string or column object inputs.

learn to implement prioritized sorting on a spark dataframe using a custom level order (beginner, intermediate, advanced) and descending ratings, via when/otherwise or expr for case when logic.

Learn to perform total and grouped aggregations on spark dataframes using groupby, applying by key aggregations to department salary expenses and revenue per category.

Validate datasets for aggregations by reading JSON files into Spark dataframes (orders and order_items) and previewing their schemas to confirm structure.

Explore common spark aggregate functions such as count, sum, min, max, and average in pyspark.sql.functions. Learn to apply them on an orders dataframe, including group by and agg usage.

Learn to perform total aggregations on a spark dataframe by filtering order_items by order_id, summing order_item_subtotal for revenue, and counting quantities with aliases.

Learn how to obtain a spark dataframe count: use the dataframe count action or pyspark.sql.functions.count in a select, and understand when execution is triggered.

Explore group by in Spark DataFrame to perform grouped aggregations or bykey aggregations, apply aggregate functions on numeric columns, and review examples and behavior with non-numeric fields.

Explore performing grouped aggregations on a Spark dataframe using direct functions like sum, min, and max, and the agg method, with practical examples on order_items and orders.

Master grouped aggregations on a Spark dataframe using agg, dicts, and column objects to compute sum, min, max, and rounded totals with clear aliases.

Master joining Spark dataframes in Spark 3.0 by setting up datasets, exploring inner, outer, left outer, right outer, full outer, broadcast, and cross joins, and applying join functions.

Explore setting up three datasets—courses, users, and enrollments—to perform joins and model many-to-many relationships with a bridge table linking users and courses in Spark.

Explore how to perform inner, left, right, full outer, and cross joins with spark dataframes using the join function, including join conditions, on clauses, and various syntax styles.

Define aliases for spark dataframes with the alias function and apply them to top-level dataframes. Use shorthand aliases to refer to columns in downstream APIs and joins.

Learn how to perform inner joins on spark dataframes to combine user and course enrollment data, project fields, use aliases, handle ambiguous columns, and count enrollments per user.

Apply left outer join between Spark data frames users_df and course_enrolments_df to combine all user details with optional course data, using projections, filters, and aliases to handle nulls.

Explore right outer joins between Spark data frames, contrast left versus right driving frames, and recognize that downstream transformations and aliasing behave consistently with full outer join semantics.

Master left outer and right outer joins on spark dataframes by choosing the driving data frame, handling one-to-many relationships with course_enrolments, and using the syntax for users, courses, and course_enrolments.

Perform a full outer join between two Spark dataframes, analyze overlapping and nonmatching records, and use coalesce and aliases to prioritize data from the first dataframe while handling nulls.

Explore broadcast join in Spark, including map-side and replicated joins, and compare with reduce-side join; learn about auto broadcast threshold and how to disable it with zero.

Use the crossJoin function on spark dataframes to create a Cartesian product, as shown with courses_df, users_df, and course_environments_df, yielding 50 records from 10 and 5 inputs.