Mastering Databricks & Apache spark -Build ETL data pipeline

Explore etl basics by extracting data from multiple sources, centralizing it in a landing region, transforming it, and loading into a data warehouse, data lake, or for stakeholder analysis.

Create an Azure storage account and container, upload data files, then mount the storage to a database using a shared access signature and connection string to access the files.

Learn to add a schema to data files in a notebook, define employee id and salary as integers, and attach the schema to a dataframe for display.

Create a notebook and database, then define an unmanaged table on the data files. Dropping the table leaves the files intact, preserving the data while the schema is maintained.

Explore managed tables in Databricks, creating a table, adding comments, and moving data from unmanaged to managed, with files stored in user hive warehouse and deleted on drop.

Apply window functions to compute maximum salary by country, then refine by gender using partitions for Canada and USA.

Learn Scala dataframe filtering in Spark: filter department equals 1 with $"dept" === 1, use not equals, and filter by first name Hugo.

Learn to perform common Scala data frame operations in Spark: select columns, rename and drop columns, add computed columns, cast types, and verify results for building ETL pipelines.

Learn Scala aggregation commands to compute max, min, average, mean, and sum, including distinct counts, count employees per department, and sort results by department id in descending and ascending orders.

Apply window functions in Scala to compute max and min salary per country, and explore rank, dense rank, and row number scenarios using partition by and order by.

Discover how to filter dataframes in spark with a custom schema, using equals, not equals, greater than, less than, and is null checks to refine department data for etl pipeline.

Master common Python operations on data frames, such as selecting and dropping columns, renaming, changing data types, and adding new columns, in a Spark-based ETL workflow.

Learn to perform data aggregation in python by building a notebook, creating a custom schema, and a data frame, then compute max, min, mean, sum, and group by department IDs.

Learn how to use Python window functions to compute max and min salary by country on an oversample data set, and compare rank, dense rank, and row number for gender.

Explore Spark SQL common operations by creating a notebook, running select queries, using where, between, not null, and casting or renaming columns for refined results.

Explore Spark SQL aggregation commands to compute max, min, average, count, and distinct counts on dataframe with a custom schema. Sort results by department id in ascending and descending orders.

Explore Spark SQL window functions to compute max and min salaries by country and assign rankings with rank, row_number, and dense_rank using partition by country and salary order.

Create a data frame with the customer schema, then build a global temporary view in spark sql and access it from another notebook using a qualified database name.

Create and query temporary views in Spark SQL inside a notebook, demonstrating that they are session-based and vanish when the session ends, and cannot be accessed from another notebook.

Explore different kinds of joins (inner, left, right, full) to combine tables A and B on department I.D., and understand how each join affects the resulting rows.

Learn how to perform left, right, and full outer joins in Scala on Spark dataframes, create temporary views, and run Spark SQL queries to combine department data and handle nulls.

Explore Python joins in Spark using Databricks, applying left, right, and full outer joins with join conditions and creating temporary views to combine data frames.

Learn how to perform Spark SQL joins by creating a notebook, constructing data frames and temporary views, then joining employee and department tables using left, right, and full joins.

Install the Spark Dash XML package on a Databricks cluster to process a nested XML file, extract first and last names, flatten the data, and cast columns with Spark SQL.

Process JSON files by flattening the customer column and extracting first and last names. Change data types for employee_id, age, salary, and department, and create a temporary view with SQL.

Learn how delta tables support upserts through merge, track changes with a transaction log and time travel using as of versions, and perform vacuum to reclaim space.

Create landing and archive containers, process json and xml files, load into Azure SQL, extract credentials from Azure Key Vault, and orchestrate a dashboard with Azure Data Factory.

Create an Azure SQL database in the portal with a resource group and basic settings, then enable the server firewall to allow your client IP for access from Azure Databricks.

Learn to create an Azure key vault in the portal, then generate and store secrets like a database username and restoring password for the database to secure access.

Learn to create and manage secret scopes and secrets, name a scope SAC, and configure access using Azure vault details to securely connect to your secret database.

Create two storage containers named landing and archive, then mount them in the Databricks project. Set up a batch process folder and create a one-time data warehouse database named Grandison.

Process multi-format data in notebooks by reading csv, json, and xml files, casting types, and loading into delta tables and Azure SQL via JDBC, with separate notebooks for each format.

Move processed csfi and xml files from the landing container to the archive container. Build dashboards for top five expensive resorts per department and salary analytics by country.

Build an end-to-end etl data pipeline with Databricks and Apache Spark, creating a dashboard that visualizes top salaries by department, costs by country, and department salary ranges.

this lecture demonstrates orchestrating a data pipeline with Azure Data Factory, creating a linked service, configuring Databricks notebooks, triggering processing, and moving files from landing to archive while monitoring results.

Celebrate completing the Databricks and Apache Spark course and reflect on a helpful learning experience, and invite learners to post any questions.