Azure Databricks - Build data engineering and AI/ML pipeline

Explore anomaly detection concepts in data pipelines, including contextual and collective anomalies, supervised and unsupervised approaches, and threshold methods to flag unusual patterns.

Learn density based outlier detection using local reachability distance and local outlier factor, and implement a cannon based anomaly detector in Python with contamination and neighborhood parameters.

Learn how a one-class support vector machine trains on a two dimensional dataset to distinguish normal from anomalous data using fit, predict, score, enabling noise reduction and semi supervised insights.

Learn how isolation forest isolates data points to identify outliers in unsupervised settings; the algorithm repeatedly partitions features with random values to isolate points, with outliers requiring fewer steps.

Apply a cluster-based unsupervised method to detect outliers in the Leicester dataset, using contamination and perimeter settings to classify data points and reveal anomalies.

Explore the angle-based approach for outlier detection, using contamination and the number of neighbors, with two variants: k-nearest neighbors and all training points.

Explore feature bagging by randomly selecting feature subsets to create diverse base estimates and combine them with mean or max to improve detector accuracy.

Explore k-nearest neighbors based anomaly detection, measuring distance to neighbors, using average or median neighbor distances to flag outliers, and visual boundary concepts to identify anomalous observations.

Apply local outlier factor to detect anomalies in a numeric dataset. Convert categorical features to numeric, set the number of neighbors, fit with scikit-learn, and flag outliers above a threshold.

This PCA-based approach detects anomalies by projecting data onto a low-dimensional subspace spanned by major eigenvectors, then scoring samples by distance from that subspace.

Explore how a data lake enables cheap, scalable storage for data, including video, images, and free text, to support reporting and machine learning workloads in Azure Databricks.

Explore how Delta Lake, created by Databricks, enables governance and raw data ingestion through bronze, silver, and gold tables for analytics and machine learning.

Delta Lake's four components leverage parquet data in object storage, a transaction log for versioning and metadata, and a high-performance engine to enable fast, scalable queries over data lake tables.

Learn to load a pollution dataset with PM2.5 and other contaminants into delta lake, infer schema, create delta tables, and apply cleaning with mean imputation, window functions, and upsert operations.

Examine the architecture of a data engineering and AI/ML pipeline: ingest from IoT and sensors into blob storage, transform to Delta Lake, train and log experiments, and deploy the model.

Upload files to DBFS and demystify the file system to prepare data—such as bitcoin data and user sites—for use in the sportswriter streaming course.

Explore streaming concepts in Azure Databricks, configuring micro-batch processing, schema management, and output streams with Kafka or Azure Event Hubs, plus checkpoints and trigger-driven execution for low-latency, incremental data pipelines.

Explore autoloader concepts in Azure Databricks, enabling incremental data loading as it lands on cloud storage with notification triggers and checkpointing for streamlined ingestion.

Mount a dbfs file location by creating a virtual link between a blob storage container or folder and the workspace, ensuring data never leaves the cloud.

Explore the data hierarchy from top level to monthly and daily levels, and examine a three-column file where the first column holds the contamination parameter and the third holds content.

Autoloader lab demonstrates ingesting streaming data into a bronze delta table in Azure Databricks, using checkpointing, schema handling, partitioning, and column renaming, with small-file optimization.

Train an isolation forest on a preprocessed transaction dataset to detect anomalies, using feature engineering like origin/destination mappings and balance changes, achieving high accuracy.

Explore model interpretability and explainability tools that reveal why a black box model makes predictions and show feature attributions to debug and improve machine learning models.

Interpret machine learning models with Shapley value using the shap library in Python, focusing on feature contributions for both global and local predictions, including anomaly detection with isolation forest.

Explore global interpretability with Shapley values, understanding how the model as a whole explains feature importance and how individual features shape predictions locally.

Discover how Kafka provides fast, scalable messaging through topics and partitions, with brokers, producers, and pull-based consumers that track offsets and rely on replication and consumer groups.

Explore how Kafka ingests real-time Wikipedia edits through language-specific topics and partitions, with offset-based reading and Spark streaming in Azure Databricks.

Analyze Kafka log messages to count IP addresses by class in a 10-second window, visualize results on a world map, and stream data with checkpointing and Delta Lake storage.

Consume Twitter data from Kafka in a streaming pipeline, define a schema, extract hashtags, count and group by hashtag, and map country information to visualize trends.

Understand tracking for machine learning experiments by organizing runs with parameters, metrics, artifacts, and source code, and how a server hosts many experiments for reproducibility.

Demonstrates MLflow tracking for training and outlier detection, showing how to log experiments, parameters, and artifacts while preparing data and evaluating models.

Explore a centralized model registry that manages the full lifecycle of ML models, from registration and versioning through staging to production, with state transitions, deployment controls, and governance.

Demonstrates deploying new model versions via a model registry, performing state transitions from staging to production, and managing artifacts and versions for factor models in an ai/ml pipeline.

Explore how to build and deploy ML inferences using MLflow on Azure Databricks, comparing Python function-based and Spark-based flows with Apache Arrow optimized data transfer.

Train ml models in parallel with pandas udfs in Azure Databricks by building a dummy regression dataset with devices and sensors, applying Spark udfs, and evaluating with mse.

Discover how applyInPandas() applies a Python function to each group, using a schema to label and predict outputs within a Databricks data engineering and ai/ml pipeline.

Learn how to productionize a machine learning model in parallel by preparing features and target variables, training and evaluating, deploying to production, and monitoring predictions in an Azure Databricks pipeline.