Building AI Applications with Databricks and Gen AI

Create schemas within a Databricks workspace and organize data into bronze, silver, and gold layers, while configuring catalogs, volumes, and user permissions.

Understand how to create and manage user groups for access control in Databricks, assign roles for developers, admins, and end users, and grant catalog and schema permissions using groups.

Learn to access an S3 bucket from AWS Databricks, set read-only IAM permissions, create a secret scope for keys, and securely read CSV data with Spark.

Ingest data from an S3 bucket into the bronze branch layer using Databricks Auto Loader, configure AWS credentials and schema, and parameterize the workflow for dev and prod catalogs.

Set up a git repository in Databricks by linking GitHub, creating main and develop branches, organizing raw and one-time folders, and committing notebooks and workflows to version control.

Learn to ingest all other data into the bronze layer by parameterizing a notebook and setting up a parallel workflow across six tables.

Learn to clean branch data, profile quality, and manually build a lake house quality dashboard in Databricks by monitoring data health and creating dashboards.

Automate creation of data quality monitors for all bronze tables using the Databricks SDK, configuring schedules, catalogs, schemas, and dashboards to ensure ongoing data quality.

Create a clean notebook to transform patient data, standardize gender, fix dates, and load into the silver layer from the bronze lake house. Use incremental merge to avoid full reloads.

Demonstrates building an incremental load workflow in a clean notebook, using a silver table merge on patient_id to update or insert records, and optimizing multi-table medical data pipelines.

clean and standardize multiple notebooks in Databricks by cloning, renaming tables, and converting Fahrenheit to Celsius in PySpark, with date formatting and table cleanup.

Design a workflow to clean silver tables by triggering the silver clean notebook. Use catalog_name and schema_name parameters, clone notebooks for the gold layer, and enable vector search after ingestion.

Create a master patient dimension in the gold layer by joining six tables with left joins, removing source columns, and saving as master_dim_patient_info to support vector search and embeddings.

Create a vector search endpoint in databricks by building a merged patient data column, generating embeddings, and indexing a vector search index to enable a similarity-based chatbot.

Register a Hugging Face embedding model into the gold catalog and deploy a serving endpoint using MLflow and SDKs, validating the model before use.

Learn to generate embeddings from a table, merge columns into a description, and create a vector search index using an embedding model and embedding endpoint and a delta-enabled pipeline.

Set up a Databricks workflow with notebooks and embeddings, query a serving endpoint for top predictions, and build a vector search index with embedding models using LM foundation models.

Create a vector search endpoint in Databricks by preparing data, embedding, and a vector search index, then register and use a foundation LM model to answer queries with retrieved context.

Check the deployed model status via the endpoint, polling until ready or reporting failure, then query with curl, Python, or the i underscore query SQL function for a serverless chatbot.

Configure and run Databricks workflows with a service principal, ensuring role-based access and serverless compute. Build and commit workflow definitions via git for reliable deployment and execution.

Build and deploy a streamlit chatbot app using Databricks templates, configure the app with a serving endpoint and environment files, and customize prompts along with internal group permissions.