Master Azure Databricks

Explore data pipelines, from extract, transform, load to batch and streaming workflows, using Apache Spark on Databricks to manage real-time and historical data with scalable, fault-tolerant processing.

Explore Azure Databricks concepts including workspace components such as notebook, dashboard, library, repo, and experiment. Understand data management, computation, model management, and authentication with MLflow, clusters, database, and permissions.

Explore Azure Databricks architecture, detailing the data plane with dbfs on customer storage, data processing by Azure VMs, and the control plane with notebooks, clusters, and authentication via Azure Directory.

Learn to create an Azure account on the portal, verify email, complete profile setup, and choose pay-as-you-go to access database services and set up a Databricks workspace.

Learn how to create and launch an Azure Databricks workspace: configure subscription, new resource group, workspace name, region, premium RBAC, review, deploy, and sign in via Azure Active Directory.

Explore Azure Databricks module by covering workspace basics, runtimes and cluster creation, notebooks, libraries and installation, repo integration, DBFS, DB utils, widgets, workflows and jobs, and external metastable grade setup.

Explore the Databricks workspace to access assets like clusters, jobs, notebooks, and libraries; navigate panels for data science, machine learning, and SQL, and manage access, compute, and workflows.

Learn about Databricks runtimes, including standard, machine learning, photon, and lite, and how to select the right runtime when creating a cluster.

Learn how Databricks clusters work, including interactive and automated clusters, standard, high concurrency, and single-node modes, and how to configure auto scaling, runtimes, and job clusters.

Attach a notebook to the cluster, install libraries, and use the spark ui to monitor stages and driver health. Edit cluster settings and manage permissions; clone, restart, or delete cluster.

Master notebooks as a web based interface to write code in Python, Scala, or SQL, manage with UI and CLI, and enable visualization and version control.

Learn how to work with third-party libraries in Databricks by installing at workspace, cluster, or notebook scope using Python, Scala, or Java libraries from PyPI, Maven, and public repositories.

Master Azure Databricks repos' git integration by cloning remote repos, managing branches, and pushing, pulling, and committing changes across providers like GitHub, GitLab, and Azure DevOps.

Learn how the Databricks file system (dbfs) abstracts Azure blob storage into a mounted, credential-free data layer for notebooks and clusters.

Master Azure Databricks dbutils by exploring fs utilities, libraries, notebooks, and widgets to manage object storage, run notebooks, handle secrets, and streamline jobs.

Explore Databricks widgets to pass parameters to notebooks and dashboards, creating text, dropdown, combo box, and multi select widgets. Retrieve widget values via the argument method in SQL.

Learn how to create and manage Databricks jobs, running non-interactive code on a cluster with notebooks, jars, or pipelines, including scheduling, parameters, alerts, and dependency workflows.

Learn to set up an external metastore with SQL Server, enable cross-workspace metadata sharing, and configure clusters with metastore connections to access shared database information.

Set up and access an external metastore across multiple Azure Databricks workspaces by creating a blob storage-backed external table, mounting storage, and configuring Delta tables with shared metadata.

Outline the module agenda, covering added concepts, transformations, actions, joints, adt, broadcast variables, accumulators, converting adt to dataframe, and importing data, creating tables via UI or notebook.

Explore Spark fundamentals with RDD, DataFrames, and Datasets. Learn in-memory, immutable, fault-tolerant data structures, partitioning, and lazy transformations with map, filter, and count.

Create adt using parallelize, textFile, and wholeTextFiles with a spark context, inspect partitions, and demonstrate empty adt with and without partitions, then collect and display contents.

Explore key rdd transformations in Azure Databricks lab, including map, flat map, filter, distinct, union, intersection, map partition, zip with unique id, repartition, and coalesce, with hands-on examples.

Master RDD actions in Databricks: learn collect, count, take, top, first, reduce, save as text file, minimum, maximum, by value, and for each operations to return results from a dataset.

Master Azure Databricks by exploring RDD joins, including inner, left outer, right outer, and full outer joins, using employee and department examples to illustrate common IDs and outputs.

Explore pair RDDs with key-value pairs, applying group by key, reduce by key, and map values, then perform count by key and lookup to see aggregation results.

Use broadcast variables to cache lookup data on all worker nodes, avoiding shuffles and speeding up email lookups from employee id within a dataframe.

Explore how accumulators serve as global counters in a spark application, updating on the driver node as partitions process records, demonstrated in Databricks with a sales.txt file.

Convert rdd to DataFrame by parallelizing a Python list, define a schema with first_name, last_name, vendor, salary, and use DataFrame methods to display the resulting DataFrame.

Upload CSV files to DBFS, then read them with Python CSV and the Spark API in a Databricks lab session.

Create a table with the Databricks UI by uploading a CSV from DBFS, configuring the header and schema, and creating it in the default database.

Create a table from a csv in dbfs using a notebook, auto-generating code to build a dataframe, register a temporary view, and save as a permanent parquet table.

Master Azure Databricks introduces dataframes, showing how to create and define schemas, perform updates, ordering, grouping, deduplication, apply casting, windowing, joins, date and timestamp functions, and user defined functions.

Explore what a DataFrame is in Spark, its advantages over ADT, including custom memory management with Tungsten and an optimized execution plan for reading and writing various formats.

Learn to create dataframes in Databricks using multiple methods, including csv, text, json, avro, and Kafka sources, and to read and write data with defined column names.

Define and inspect dataframe schemas in Databricks by retrieving column names and data types, and create a new schema using StructType and StructField for precise typing.

Explore Databricks functions to manipulate dataframes: select specific columns, create new columns with withColumn, concatenate, rename, drop, and apply when/otherwise conditions in Spark dataframes.

Explore casting operations in Spark dataframes by converting string columns to boolean, integer, double, and date using withColumn, selectExpr, cast, and SQL, including salary.

Use the filter transformation in Databricks to extract dataframe records by conditions such as equal, not equal, greater than, starts with, ends with, and null checks.

Explore union and union all to merge two dataframes with the same schema into a single dataframe, then use union by name with allow missing column to merge different schemas.

This lab demonstrates ordering data with sort and order by in Azure Databricks, arranging the department column in ascending or descending order, and applying multi-column and temp view methods.

Group by operations cluster identical data by department in a dataframe, then compute maximum, minimum, and average salaries, and count records for each group.

learn to remove duplicate records from a dataframe using distinct and drop duplicates, including removing by a specific column like employee name in a notebook example.

Explore window functions and analytic functions that operate on related rows using partition by and order by in spark and databricks. Learn row_number, rank, dense_rank, percent_rank, lag, and lead.

Explore date and timestamp functions in Azure Databricks in this lab, learning to add months, current date, date formats, date arithmetic, and conversions between strings and dates.

Create a Python function, register it as a UDF in Databricks, and apply it to dataframes to capitalize first letters and lowercase the rest.

Master Azure Databricks joins by combining employee and department data with inner, left outer, right outer, full outer, left semi joins, and self joins, while handling duplicate columns.

Handle bad data in Databricks using Apache Spark and the badRecordsPath option. Apply a schema to exclude invalid records and inspect the bad data folder for the two bad entries.

Learn how to use the underscore metadata column to fetch file_path, file_name, file_size, and file_modification_time for input files in Databricks, across formats like CSV, JSON, packet format, and XML.

Explore reading and writing CSV, XML, and Excel files, learn to read and write to SQL databases using GBK protocol, and access data in Azure Blob Storage via mounting.

Learn to read and write parquet data in databricks, define a schema for a csv-derived data frame, and save, append, or overwrite partitioned parquet files by country.

Demonstrates reading a customer CSV from DBFS with header and a defined schema, handling nulls via permissive, drop, and fail-fast modes, and writing the CSV back to storage.

Learn to read single-line and multi-line JSON files in Azure Databricks, handle nested structures, and flatten complex JSON into a clean tabular dataframe.

Read and write XML files in Azure Databricks with the Spark-XML library, converting simple and nested XML into a dataframe and exporting back to XML.

Learn to read and write excel files in azure databricks by uploading an all users excel to dbfs, installing a maven library, reading multiple sheets, and saving back to excel.

Learn to read and write data from a SQL Server using JDBC in Databricks, configure the driver and connection options, manage partitions, and overwrite or create tables.

Learn to read and write files in Azure blob storage, create containers, upload data, and mount blob storage in a database or notebook using access tokens and Spark settings.

Explore structured streaming concepts. Build a streaming ETL with Kafka sources and Event Hub integration, apply transformations, and ingest data into storage, then set up Twitter sentiment analysis.

Master Azure Databricks covers Spark structured streaming, unbound and input tables, and unified APIs for batch and streaming data, with watermarking, triggers, output modes, and fault-tolerant checkpoints.

Learn about data sources and sinks in Spark Structured Streaming, including file, Kafka, socket, and rate sources, and file, Kafka, and memory sinks.

Generate streaming data with the rate source by setting rows per second and producing timestamp and value columns, then load file source data from csv files into a streaming dataframe.

Set up Kafka on Linux, start Zookeeper and Kafka services, create topic one, and stream data into a Databricks Spark job using Kafka format with bootstrap server and latest offset.

Master console, memory, and file sync in Azure Databricks using write stream, producing csv or json outputs and enabling for each batch sync to SQL Server.

Explore streaming ETL in Azure Databricks, processing real-time IoT data with low latency, and loading transformed data into Azure blob storage or data table via Azure Event Hub.

Learn to set up Azure Event Hub in the Azure portal by creating a namespace and an Event Hub, configuring partition count, message retention, and capture.

Learn to send data to Azure Event Hub with a Databricks producer: install the library, configure the connection, create a batch, and send a hello world event.

Learn to read Azure Event Hub messages in Databricks with the Spark Read Stream API, install the Maven library, configure encryption, and cast binary body to string for streaming.

Transform json messages into a tabular dataframe by defining a schema, casting the body to string, parsing json, and extracting fields.

Ingest real-time streaming data into Azure data storage by mounting blob storage in Databricks and export the streams as CSV, JSON, and Delta tables with checkpointing.

Explore Twitter sentiment analysis by building a streaming pipeline that uses ETL to extract and transform data from real-time and historical tweets, then classify text as positive, neutral, or negative.

Learn how to create a Twitter developer account, verify your email, and apply for elevated access, then configure authentication with client ID, client secrets, and callback URLs.

Install and configure Twitter API libraries in Databricks, stream English tweets about Kuwait, and push them to an event hub for real-time sentiment analysis.

Read Twitter data from an Event Hub in Azure Databricks and apply sentiment analysis using the Fin library to classify messages as positive, neutral, or negative, then display results.

Explore the Databricks SQL Warehouse and its components. Create tables from CSV, write parameterized queries, build dashboards, charts, and alerts, and manage access and ownership.

Understand databricks sql concepts and components: interface, data management with visualizations and dashboards, alerts, sql warehouse, query history, and authentication with users, groups, tokens, and access control.

Learn how to configure a skill warehouse in the Databricks SQL workspace, set cluster size and auto stop, enable scaling, obtain connection details and tokens, and monitor queries.

Learn to create a table from a CSV in Databricks SQL by uploading the file, auto-detecting column types and delimiter, and reviewing the header and sample data.

Create and run SQL queries on the customer table in the Azure Databricks analytics workspace, using the schema browser, save, share, format, schedule, and export results to CSV or Excel.

Learn to configure and use runtime query parameters in a lab, creating text and number parameters, dropdown lists, and date types to filter data with in-clauses and single-quoted values.

Learn to use query profiling to obtain performance metrics, troubleshoot bottlenecks, and visualize each query task's time, rows, and memory while fixing common SQL mistakes like full table scans.

Master Azure Databricks shows how to create and customize visualizations: tables, bar charts, and pie charts, using Databricks SQL with formatting, filtering, and labeling for customer data.

builds a counter chart of total employees by counting distinct customer IDs and a line chart of yearly total monthly charges in the Azure Databricks lab.

Learn to create a consolidated dashboard in Databricks SQL by integrating multiple visualizations—table, pie, bar, line, and counters—into a single interactive customer dashboard.

Learn to set up Databricks alerts that email you when a scheduled query meets a threshold, such as total employees below 8000, and how to manage them.

Learn how admins grant access to Databricks SQL objects by assigning permissions for queries, dashboards, alerts, and SQL warehouses using UI or REST API, including run as owner/viewer roles.

Demonstrate how to grant and revoke database and table permissions in Databricks, using grant options to assign select, read metadata, usage, and modify rights.

Learn to transfer ownership of SQL objects—queries, dashboards, alerts, and data—from admin to a new owner using the user interface or REST API.

Learn to read and write data tables from Python using the Databricks SQL Connector, connect to a cluster or warehouse, and run queries to insert and fetch delta table data.

Run sql queries from the terminal with the databricks sql cli. Set up a virtual environment, install the cli, and configure hostname, http path, and access token for authentication.

Explore data lake and data lakehouse architectures, including medallion design, and learn to create, read, update, delete, and merge data tables with Python.

Explore data warehouse, data lake, and the data lakehouse, a unified architecture that handles structured, semi-structured, and unstructured data for analytics, machine learning, and governance.

Show how medallion lakehouse architecture creates a single source of truth with bronze, silver, and gold layers to ingest, validate, and refine data for analytics, machine learning, and production applications.

Delta Lake on Azure Databricks adds reliable transactions and scalable metadata to data lakes, unifying streaming and batch processing, with schema enforcement, time travel, and merges for inserts and updates.

Create and manage Delta tables on Azure Databricks using SQL and Python, including managed and unmanaged tables, with partitioning by state and Delta file workflows.

Learn to ingest data into a delta table using insert into and copy into, create tables, and manage records with append and overwrite modes in Python.

Master delete, update, and merge operations in Delta Lake with the data table API and Python, using delete from, update, and merge into to handle inserts, updates, and deletes.

Learn schema validation and evolution in Delta, enforcing data quality by rejecting extra columns, data type mismatches, and case-difference column names, and enable evolution with merge schema.

Master azure databricks explores time travel in delta lake, enabling version- and timestamp-based queries to fix data mistakes, restore tables, audit changes, and support debugging and reporting.

Convert parquet tables to delta format in Azure Databricks by creating a user_packet table, running the convert to delta command, and migrating all parquet tables to delta.

Discover generated columns in a data lake, auto computing derived fields like date of birth via user-defined functions and business logic, with table creation, inserts, and Python usage.

Build an incremental ETL load pipeline in Delta by merging SQL Server data into a Delta table, tracking last modified dates in a control table, and scheduling a notebook job.

Master incremental etl with stable versioning by performing a full load, merging changes, tracking last version in a control table, and computing delta differences with anti join and except all.

Explore how to manage users and groups, add users to Active Directory, and assign them to groups. Learn to apply table access control and grant workspace, cluster, and job access.

Explore Databricks identities—users, groups, and service principals—and how account admins and workspace admins manage users, groups, permissions, and workspace settings.

Learn to add a new user to Databricks by creating an Azure portal account, inviting them to azure active directory, and granting databricks access for login.

Create groups in Databricks and assign users to developer and data science groups, then configure entitlements such as space access, unrestricted cluster creation, and database scale access.

Enable and manage table access control in Databricks, programmatically granting and revoking data access for users via Python and SQL, with cluster-level settings and permissions.

Enable workspace, cluster, and job visibility controls in Databricks and assign permissions like can start and can manage for notebooks, clusters, and jobs to users or groups.