Learning Azure Databricks

Explore Azure account essentials for Databricks, including free trial setup, pay as you go, quotas, and vcpu considerations for compute on virtual machines.

Learn how to create an Azure free trial account on portal.azure.com, get $200 credit for 30 days, and understand transitioning to pay-as-you-go with manual billing and verification steps.

Compare the free trial and pay-as-you-go in Azure Databricks, noting 750 hours of b1s linux compute and limited sizes in the free tier, versus broader DS3 v2 options in pay-as-you-go.

Learn how to create a pay-as-you-go subscription in Azure for Databricks practice, including getting started, confirming terms, entering payment details, and receiving a paid subscription by email.

Explore the Azure portal (portal.azure.com), sign in, browse all services to locate Azure Databricks, and use Cloud Shell with Bash or PowerShell to list storage accounts via Azure CLI.

Increase DS v2 vm-family vcpu quotas in Azure Databricks by navigating to quotas in portal.azure.com, ensuring you have contributor or owner permissions, and submitting a regional quota request.

Learn to set up budgets in the Azure portal to monitor pay-as-you-go billing, create thresholds, and receive email alerts during monthly cycles.

Lower Azure Databricks costs by setting budgets and alerts, turning off unused resources, and using serverless data warehousing along with repos or drives and stop-start practices.

Explore Apache Parquet, a fast, open, columnar storage format favored for efficient compression and query speed on Databricks and the Hadoop ecosystem. Showcases faster analytics and lower costs versus CSV.

learn to create Azure Blob Storage and Azure Data Lake Storage Gen2, enable hierarchical namespace, and manage parquet data for Azure Databricks workflows.

Learn to create an Azure Databricks workspace by selecting a resource group, naming the workspace, choosing East US and premium tier, configuring encryption, tags, and review before creation.

Learn how to create users in an Azure Databricks workspace, assign permissions via Azure Active Directory and access management, and verify login with AD credentials.

Learn how to create and register a service principal in Azure Active Directory, grant Databricks access to data lake storage, and manage permissions for automated workloads.

Launch the Azure Databricks workspace, create a compute cluster, and configure a multi-node or single-node Apache Spark runtime with auto-delete after 60 minutes to prevent idle billing.

Learn to generate and manage tokens in the Azure Databricks workspace by using developer settings, generating a token with a set validity, copying and securing it, and revoking when finished.

Create your first Azure Databricks notebook and run magic commands across Python, SQL, Scala, and R. Use markdown, HTML, and display features in notebooks.

Learn how to integrate GitHub repos with Azure Databricks by linking accounts, authorizing access, and syncing notebooks through a Git workflow.

Explore Databricks utilities to enable programmatic access to notebooks and the Databricks file system, and practice using widgets and basic data engineering resources.

Explore how to navigate and use the Databricks file system (dbfs) with Databricks Utilities, listing files, creating directories, and mounting and copying data within a workspace.

Learn how to create a directory in the Databricks file system (dbfs) using dbutils, verify with a listing, and prepare for copying data in the next lecture.

Learn how to upload a file into Azure Databricks, create a volume, copy a parquet file using Databricks Utilities file system, and verify the file location.

Learn how to delete a file or directory using Databricks utilities by listing the file system, uploading a file, and removing it with dbutils.fileSystem.remove.

Explore how to use Databricks Utilities to orchestrate notebooks, run companion notebooks with dbutils notebook run, manage timeouts, and view cross-notebook outputs.

Explore Databricks widgets by building a drop-down menu to select a language (Python, SQL, al, Scala, and R), then retrieve and print the selected option using Databricks utilities.

Create an azure data lake storage gen2 in the portal, enable the hierarchical namespace, upload the New York taxi dataset parquet file, and connect it to azure databricks for processing.

Connect azure data lake storage gen2 to azure databricks using the wasbs driver via a notebook, configuring storage account, container, and sas token.

Connect to your Azure Databricks notebook, create a Spark data frame named green taxi data frame, then read parquet data from mounted data lake storage and display to verify.

Connect Azure data lake storage in Azure Databricks using the ABFSS driver and mount the storage with account, container, path, and key, validating access with a taxi parquet dataframe.

Register a data frame as a temporary view and run SQL queries to compute min, max, and average fare amounts, then explore Delta Lake for persistent table storage.

Learn why Delta Lake tables convert Apache Parquet data in Azure Data Lake Storage into ACID-compliant tabular Delta Lake tables for SQL data warehousing in Azure Databricks.

Create Delta Lake tables in Azure Databricks by loading a taxi dataset into a data frame, writing it in Delta format, and saving as a Delta table, then read it.

Read data from Delta Lake tables into a Spark DataFrame in Azure Databricks by loading from the Delta format at a given path.

Apply conditional updates and overwrites to a Delta Lake taxi table, verify Delta Lake table status, and load the updated data using the delta format.

Explore time travel with Delta Lake tables by viewing full history, listing versions and timestamps, and rolling back to a chosen version to revert data changes.

Apply delete operations on Delta Lake tables using Spark to remove rows by a timestamp condition. Verify the deletion by reading the table before and after, and comparing row counts.

Learn how to drop a Delta Lake table in Azure Databricks using Databricks utilities, verify the table with a data frame, and confirm the path no longer exists.

Discover data warehousing on Azure Databricks by mastering ingestion, processing, storage, modeling, querying, reporting, and governance with medallion architecture, ADLS, and role-based access control.

Model data with the medallion architecture in Azure Databricks, transforming raw data through bronze, silver, and core layers. Deliver gold datasets in Delta Lake tables for SQL queries.

Create a serverless SQL data warehouse in Azure Databricks by editing the starter warehouse, selecting serverless compute, and enabling pay-as-you-go start-stop usage for on-demand use.

Explore how the metastore, catalogs, and schemas organize tables within a serverless SQL data warehouse in the Azure Databricks workspace, spotlighting the catalog explorer and default schema.

Explore the data ingestion methods in Azure Databricks, including ad data UI, copy into, and autoloader, pulling data from Azure Blob Storage or Data Lake Storage into Delta Lake tables.

Learn how to create a managed volume in Azure Databricks by using the catalog tab and catalog explorer, name the volume, and prepare to upload files.

Ingest data into Azure Databricks using the Add Data UI method, build bronze, silver, and gold layers and delta tables, and query in the SQL data warehouse.

Learn how to use the copy into method to load data from file locations into Delta Lake tables, supporting CSV, JSON, parquet, and other formats from cloud storage.

Use copy into to ingest data into Delta Lake, build bronze, silver, and gold layers in a medallion architecture, and load employee dot csv and department dot csv into bronze.

Explore ingesting data in Azure Databricks across bronze, silver, and gold layers. Use copy into to move joined silver Parquet data into Delta Lake tables, with schema merge enabled.

Use copy into to ingest new records into delta lake in azure databricks, propagating changes from bronze to silver to gold layers for querying.

Ingest data into Azure Databricks with autoloader to populate bronze, silver, and gold Delta Lake tables, then transform and join employee and department data.

Explore Unity Catalog, a cloud-agnostic, centralized governance solution that manages data assets and permissions across lakehouse environments from a single metastore, with catalogs, schemas, tables, and volumes.

Learn to set up unity catalog by provisioning two azure databricks workspaces in east us with premium tier and joining them under the catalog.

Resolve Azure Databricks management login errors by creating a separate global administrator user, assigning admin roles, enabling MFA, and using this account to access accounts.databricks.net.

Learn how to resolve access issues to Azure Databricks workspaces by configuring a global administrator, granting workspace permissions, and validating access across workspace one and two.

Learn how to access and manage two Azure Databricks workspaces, enable unity catalog, and prepare a unified catalog across workspace one and workspace two, including storage, metastore, and table sharing.

Create and bind a new storage account for unity catalog by granting two azure databricks workspaces—via system-assigned managed identities—the storage blob data contributor role, using two access connectors.

Create a custom metastore for Unity Catalog in East US, configure the ADLS Gen2 path, and attach it to workspace one and two, then enable Unity Catalog.

Explore Unity Catalog sharing across two workspaces by provisioning a main metastore, creating a data volume for employee and department data, and building bronze, silver, and gold layers.

Explore how Apache Spark structured streaming processes real-time data with Spark SQL, delivering scalable, fault-tolerant micro-batch processing and incremental data handling.

Explore how Apache Spark handles Structured Streaming with micro batches and continuous processing, revealing latency down to one millisecond and fault tolerance from exactly once to at least once.

Learn the programmatic approach to structured streaming with Apache Spark, linking input tables, result tables, and outputs to an external sink, and observe updates every second.

Explore the programmatic write modes in Apache Spark structured streaming, including complete, append, and update modes, and learn how input tables, result tables, and output sinks drive word count outputs.

Create data frames from CSV files in Databricks, verify their availability, and assess streaming readiness; learn that batch data frames require readStream and writeStream APIs to enable streaming.

Learn structured streaming on Azure Databricks by initializing session and streaming context, connecting to sources like Event Hubs or Kafka, and writing to Delta Lake, Blob Storage, or Cosmos DB.

Demonstrates handling structured streaming data on Azure Databricks using Spark API, sets up a cluster, and implements the bronze–silver–gold medallion architecture with employee and department CSV data.

Learn to process streaming data on Azure Databricks by joining bronze employee and department data frames on department ID and writing to parquet in the silver layer with overwrite.

Explore the read stream API in Azure Databricks to read parquet streams with a provided schema and write to a Delta Lake table, leveraging Delta Lake for reliable streaming.

Explore the writeStream API to write a streaming data frame into a Delta Lake table, using append mode and a checkpoint location in the silver layer.

Process streaming data from Delta Lake tables in the silver layer, and compute the average salary by department using Spark SQL, with results displayed.

Explore change data capture from streaming delta tables, read updates to an employee csv, stream into bronze, write to delta lake, and verify results via sql editor.

Learn how to stop a streaming query in azure databricks by issuing a stop command, observe the spark job conclude, and understand how query await termination behaves.