Databricks Fundamentals & Apache Spark Core

Navigate the Databricks workspace user interface, create a cluster with a Databricks runtime for Apache Spark, and attach notebooks to run Spark code for data cleansing and machine learning.

Apache Spark runs on a cluster managed by a cluster manager and node managers, with a Spark driver requesting executors to run tasks across the cluster.

Create a Spark cluster and notebook, then load customer.csv into a dataframe using Spark session with read.csv or read.format('CSV').load, and display the dataframe in Databricks.

Understand how a dataframe's schema defines column names, data types, and nullability, view it with print schema, and explore the struct type of fields with name, data type, and nullability.

Filter out invalid birth records from a customer data frame using Spark's filter and where methods, handling nulls, valid ranges, and equality checks to clean the data set.

Learn to perform an inner join between dataframes in Spark by defining a join expression, joining left and right datasets, and selecting useful columns for a readable result.

Group data by city (and salutation) with the group by method to create groups, then count customers in each group, using Apache Spark.

Compute the total extended sales price per item for manufacturer 128 in November by joining date, store sales, and item data, then group by year, month, brand, and item.

Explore filtering customers born in January in Canada, using where with equality and inequality, and normalize strings with lower and trim to handle case and spaces.

Learn SQL null handling by using is null and is not null predicates to filter rows, avoiding equality checks with null, and applying these concepts to real tables.

Explore SQL aggregations with count, sum, avg, mean, min, and max, including null handling and grouping concepts using a customer table and store sales data.

Explain how the right outer join returns all rows from the right table and fills left nulls. Highlight data cleaning for invalid customer IDs before reporting.

Describe building a Databricks notebook to report total web sales for selected items and zip regions in 2001 Q2, using joins, filters, and a bar chart.

Learn to work with boolean expressions in Apache Spark to filter dataframes using where or filter. Create boolean columns and combine conditions with and and or to refine results.

learn to work with datetimes in apache spark by converting strings to dates with to_date, using lead, and computing date differences with date_diff and month_between.

Explore arrays in Apache Spark by building a data frame of customers and their item lists, and use collect_set, explode, and array_contains for analysis.

Create and query a map column in Apache Spark with category as key and product name as value, using trim, then explode and rename to category and product name.

Learn to read CSV files with the DataFrameReader in Spark, configuring format, schema, and header options. Manage malformed records with modes such as permissive, fail fast, and drop malformed.

Learn to read JSON files with Apache Spark using the Spark session data frame reader, handling single-line and multi-line JSON, and applying explicit schema, date format, and nested field queries.

Use the dataframe writer to save a dataframe to csv or json with options like header, separator, and compression, and partition by year and month.