Crack Databricks Generative AI Engineer Associate Exam

Explore how generative AI builds on AI, ML, and deep learning to generate images, text, and audio from data, using models like transformers and GANs to solve real-world business challenges.

Explore the full spectrum of generative AI applications and large language models, from image, video, 3d model, and music generation to synthetic data, design, architecture, and urban planning use cases.

Learn how large language models work: tokenize prompts, convert tokens to embeddings, and use a transformer to predict the next word with probabilities.

Explore Databricks generative AI offerings from data preparation and model serving to governance via Unity Catalog, with mlops, vector databases, and Delta Live Tables.

navigate the databricks platform, set up a free workspace via express or cloud account, and use the playground to test ai models and run experiments.

Select a model in Databricks and serve it to create an endpoint. Access the endpoint from a Jupyter notebook using the OpenAI library.

Master prompt engineering by crafting precise prompts; apply four key parameters: style, audience, length, tone; leverage the role-task-format, CTF, and Raas framework to optimize model performance.

Build and test prompts in a Databricks notebook by installing OpenAI, creating reusable functions, and crafting prompts using delimiters for summaries, JSON outputs, and multi-task workflows.

Explore retrieval augmented generation (rag) and learn to use a vector store with embeddings, indexing, and retrieval to augment prompts for accurate, domain-specific answers.

Trace a step-by-step rag pipeline: upload data to storage, use unity catalog with mosaic vector search, create chunks and embeddings in a vector database, and answer with lmh and lmdh.

Ingest data from local, S3, or Azure as tables or volumes, then parse, clean, chunk text with overlaps, build vector search index, generate embeddings, and serve via model serving.

Understand Unity Catalog as a single storage for data and AI entities, including volume, tables, embeddings, and models, with governance features for secure data sharing.

Explore the Unity Catalog in Databricks, navigate workspaces and schemas, create a volume, upload files, and set up a table with embeddings or a vector search index.

Explore solving data fiduciary problems with a Rag model to extract the Indian Data Privacy and Data Protection Act definition from a document, using the Databricks Llama 370 billion model.

Extract data from documents in the unity catalog volume via document processing and build a delta table with spark, supporting csv, parquet, text, pdf, and json.

Extract data from pdfs and other formats by loading a file path, volumes, and pi pdf, reading content page by page for rag model workflows.

Transform data by cleaning and standardizing it: remove nulls and irrelevant lines, trim spaces, extract structured fields with regex, add new columns, and harmonize currency formats across CSV files.

Explore chunking strategies, including fixed-size, paragraph, and sentence-based chunks, and embed each chunk before storing embeddings in a delta table or vector store.

Learn to chunk large documents using a recursive character text splitter, setting chunk size and overlap to produce 165 chunks from a PDF and build a delta table.

Create a dataframe with an id primary key and save the chunked content as a delta table in Unity Catalog, enabling embedding workflows.

Learn how embeddings convert product categories into numerical vectors to enable context-based search in vector db, using one-hot encoding, tag-based features, and distance metrics like Euclidean, cosine similarity, and Manhattan.

Determine embedding models by evaluating data properties, language, vocabulary, and domain, then compare model capabilities, dimensions, privacy, cost, and licensing in the Databricks ecosystem.

Learn how vector databases store high-dimensional embeddings and enable fast similarity search for rag in generative AI, with scalable indexing and options for kNN and approximate nearest neighbor searches.

Explore mosaic i vector search in databricks, a vector database for embeddings and metadata that integrates with unity catalog for governance and real-time searches via REST or Python.

Create a vector search index in Databricks by supplying endpoint name, index name, source table, embedding column, and primary key, via code or the UI.

List your vector stores with the client.list_endpoint, then list the indexes with client.list_index and fetch the exact index using client.get_index by providing the catalog schema index name.

Fetch the index and run a similarity search by providing your query, the index columns, and the number of results; top three matches come from 165 chunks.

Demonstrates using index similarity search with a query text to fetch top chunks and their similarity scores, from 165 database chunks, while noting readability and context trade-offs.

Rerank the vector store similarity results to improve precision, using algorithms like Cohere, Gina, and BGI to select the most relevant documents as context for the LLM and reduce hallucination.

Combine retrieved chunks from similarity search with the question to form a context-rich prompt, then pass it to a llm via Databricks using rag, noting upcoming chains and prompt templates.