Microsoft Azure Databricks Administration - ETL Workflow

Mount Azure Blob Storage in Azure Databricks using dbutils.fs.mount to load and save data, configure a container and SAS token, and handle mount errors.

Apply try and catch in a scala databricks ETL workflow to gracefully handle existing mounts, unmount when needed, and auto-check databricks file system mounts with dbutils.

Automate mounting azure blob storage in Databricks with Terraform by creating a secret scope and blob key, then apply to bind the mount to the storage account.

Read a travel dataset from Azure Blob Storage using Spark with Scala, infer schema, select and rename destination and original country, and apply simple transformations.

Transform csv data from azure blob storage by renaming columns, selecting necessary fields, and filtering with sql, then create a temp view and group by origin country and destination country.

Push transformed data to Azure blob storage by running a Spark SQL query and writing the result as CSV to the custom mount. Preserve headers.

Define a manual Spark schema for a heavy NYC fire department CSV with 30+ columns, then load into a dataframe with header and preview the table in Databricks.

Explore Databricks tables and the file system by creating databases, uploading CSV files, previewing schemas, and transforming data, then save results to Azure Blob Storage or a database.

Learn spark data frame transformation and action in Databricks, using select, filter, where, aggregate count distinct, and group by with order by to analyze call types.

Rename columns for data uniformity using the columnrenamed function, then apply min(), max(), and average aggregations to summarize delay and response metrics in a Databricks ETL workflow.

Learn structured streaming with Spark SQL, reading data as a stream, writing to console or other sinks, and using a checkpoint location for fault-tolerant, continuous data processing.

Create secret scopes to securely store credentials for data sources in Azure Databricks. Manage secrets using ui, cli, data form, and terraform, with Azure Key Vault or Databricks Key Vault.

Learn to secure Azure Databricks secrets using key vault, create secret scopes, and access them with dbutils.secret.get, while automating with the command-line interface and terraform for production-ready pipelines.

Automate secret management and resource provisioning with Terraform for Azure Databricks, creating storage accounts, containers, a key vault, and secrets, and binding them as secret scopes.

Learn to enable and access Azure Databricks logging—Spark UI, event logs, and driver logs—to monitor clusters and diagnose application issues, with log analytics in focus for real-time insights.

Configure an Azure log analytics workspace to stream Databricks logs, run log queries, and set up alerts, then prepare prerequisites like Docker and Maven for the monitoring library.

Configure logging in azure databricks with spark listeners and log analytics jars. Stream logs to a log analytics workspace using the azure monitoring library and init scripts.

Create a spark monitoring cluster, apply the init script, and enable log analytics to surface spark monitoring logs and errors using the coastal query language.

Automate Azure Databricks logging with Terraform by provisioning a log analytics workspace, enabling cluster logging, and deploying an init script; apply changes and validate log propagation.

Enable and configure git integration in Azure Databricks to connect notebooks with GitHub or other providers, generate a personal access token, and commit notebook changes to the remote repository.

Enable continuous integration and deploy artefacts from GitHub to Azure Databricks clusters using Azure DevOps pipelines, building artefacts and promoting to dev, non-prod, and production environments.

Configure continuous delivery for Databricks using Azure DevOps templates. Build and deploy notebooks from the drop folder artifacts to a Databricks cluster via a release pipeline, with token authentication.

Enable nightly ETL automation by running a notebook as a scheduled Azure Databricks job, reading from blob storage, transforming data, and writing results to blob storage or Databricks storage.

Enforce cluster policy and permissions in Azure Databricks to secure job clusters, defining a JSON policy that limits runtime versions, machine types, and min/max node counts.

Create an Azure Data Factory pipeline to run Databricks notebooks, configure a linked service for authentication, and orchestrate Spark jobs and transformations from Databricks.

Orchestrate notebooks with Azure Data Factory by building pipelines in a code-free UI, selecting linked services, and running notebooks from the canvas, including a streaming app that runs until stopped.