
Unify data preparation, feature engineering, and model training on delta tables with AutoML in Databricks machine learning, enabling tracking with MLflow, model registry, and feature store for deployment.
Explore creating a Databricks community edition workspace, including account setup, email verification, and choosing Azure, AWS, or GCP backends while noting compute and storage limitations.
Walks through setting up a Databricks workspace in the Azure cloud, with a pay-as-you-go subscription, creating a new resource group, and selecting the premium East US tier.
Verify Databricks workspace is ready in Azure portal, then sign in via Azure Active Directory single sign-on and explore the new UI across data science, engineering, SQL, and machine learning.
Explore the Azure Databricks architecture, detailing the control plane vs the data plane, and key components like cluster manager, UI, DBFS, vnet, storage under Azure Resource Manager and pay-as-you-go billing.
this lecture shows that an azure databricks workspace automatically creates resources such as DB manage identity, storage accounts, a network security group, and a virtual network, with potential billing.
Discover how Databricks runtime for machine learning provides a pre-built, GPU-enabled cluster with libraries like TensorFlow, PyTorch, Keras, and XGBoost for scalable model development and tracking within the Databricks workspace.
Create a Databricks ML cluster in the Azure Databricks workspace by selecting a single-node, all-purpose compute with the 12.2 ML runtime, set autopilot, and sign on via Azure AD.
Explore a ready Databricks cluster from the UI, review its configuration (version 12.2, cores, memory), and manage actions like restart, clone, delete, and permissions in Azure; terminate to release resources.
Automates machine learning on your data with Databricks AutoML, preparing data, defining prediction target, and training model trials for regression, classification, and forecasting, presenting the best model with Python notebooks.
Explore building a regression model with Databricks AutoML via the UI, starting a cluster, uploading housing.csv, creating a delta table, and training on 20,641 records to predict median house value.
Create an AutoML regression model in the Databricks UI by configuring the cluster, selecting the housing_t data, and evaluating with R2 score using sklearn, xgboost, and lightgbm.
Explore how AutoML regression in the Databricks UI evaluates 76 models, ranks them by R2 score (0.742), and analyzes parameters, artifacts, and best model selection.
Explore building an AutoML regression model with the Databricks Python API on housing data, reading a csv, configuring train/test splits, and targeting median house value.
Print experiment summaries, view best trial paths and mlflow run IDs, then generate predictions on the testing set using the Databricks Python API and evaluate with R2 and RMSE.
Explore auto ml classification in databricks, using ui and python api, with census data to predict income in a binary class, including training and testing buckets.
Explore AutoML classification workflows from Python and UI, compare models with F1 scores and hyperparameters like n_estimators, and generate predictions with mlflow, pandas, spark, and confusion matrices.
Learn to forecast Covid-19 cases with Databricks AutoML forecasting using the UI and Python API, by loading a Covid dataset into a Spark DataFrame and creating a forecasting table.
Explore AutoML forecasting with six models, evaluate their accuracy using a standard metric, and review ARIMA training, forecasts, and data exploration before implementing in Python.
Explore Databricks auto ML forecasting via the Python API, reading data into a pandas dataframe, setting date as the time column, and configuring horizon, daily frequency, and a 15-minute training.
Explore auto forecasting with Databricks Python API, load the best trial model via MLflow, and generate time series predictions. Display forecasted values alongside actual data over 30 days using matplotlib.
Learn to use the Databricks feature store to store and manage features, create a feature table from wine quality data, and add an id column for machine learning models.
Explore building and updating a Databricks feature store, training a wine quality model with real time measurement features, and performing batch scoring with versioned model deployment.
Discover mlflow, an open source platform for managing the machine learning lifecycle in Databricks, with tracking, models, registry, projects, and recipes.
Demonstrate Mlflow logging API in Databricks by creating a notebook to run experiments with scikit-learn, using diabetes data, and enabling auto logging with Mlflow 2.1.1 on a 12.2 LTS runtime.
Explore Mlflow logging API for managing runs and experiments, logging parameters, metrics, artifacts, and models in notebooks, and creating different experiments with multiple runs.
Learn to create and manage Mlflow experiments and runs with the Mlflow logging API, including default and diabetes_experiment setups, starting runs, and tracking experiment IDs.
Demonstrate an end-to-end machine learning workflow with MLflow, including model registration, production deployment, and data preparation on wine quality data from red and white wines.
Explore a wine quality dataset through Seaborn visualizations, feature importance, and model training with a random forest, including data prep, train-test splits, and MLflow logging for end-to-end modeling.
Register the model from a run id as wine quality model, promote version 1 to production, and note AUC 0.854 while exploring hyperparameter optimization with hyperopt and xgboost.
Explore end-to-end ml workflow by creating multiple models from a single run, optimize AUC via hyperparameter tuning, and promote version two to production for batch inference with mlflow.pyfunc.
Deploy a model with live inferencing by creating a one-day token, registering a wine quality serving endpoint, testing predictions via curl or Python, and cleaning up resources.
Register a forecasting model in mlflow using a wind farm dataset, log training with Keras and TensorFlow, and manage model versions in the registry.
Track and manage model registration with the databricks api and mlflow tracking client, update model and version descriptions, and stage versions from staging to production, while querying latest version details.
Load registered mlflow models, forecast with staged production versions, and transition between production and staging while comparing pyfunc, TensorFlow, and scikit-learn predictors.
Welcome to our comprehensive course on Databricks Certified Machine Learning Engineer Associate certification. This course is designed to help you master the skills required to become a certified Databricks ML engineer associate.
Databricks is a cloud-based data analytics platform that offers a unified approach to data processing, machine learning, and analytics. With the growing demand for data engineers, Databricks has become one of the most sought-after skills in the industry.
The minimally qualified candidate should be able to:
Use Databricks Machine Learning and its capabilities within machine learning workflows, including:
Databricks Machine Learning (clusters, Repos, Jobs)
Databricks Runtime for Machine Learning (basics, libraries)
AutoML (classification, regression, forecasting)
Feature Store (basics)
MLflow (Tracking, Models, Model Registry)
Implement correct decisions in machine learning workflows, including:
Exploratory data analysis (summary statistics, outlier removal)
Feature engineering (missing value imputation, one-hot-encoding)
Tuning (hyperparameter basics, hyperparameter parallelization)
Evaluation and selection (cross-validation, evaluation metrics)
Implement machine learning solutions at scale using Spark ML and other tools, including:
Distributed ML Concepts
Spark ML Modeling APIs (data splitting, training, evaluation, estimators vs. transformers, pipelines)
Hyperopt
Pandas API on Spark
Pandas UDFs and Pandas Function APIs
Understand advanced scaling characteristics of classical machine learning models, including:
Distributed Linear Regression
Distributed Decision Trees
Ensembling Methods (bagging, boosting)