End to End Azure Data Engineering Real Time Project- 3 Hours

Master the environment setup for an end-to-end Azure data engineering project, including resource group creation and Azure Data Factory, Databricks, Key Vault, Azure Storage, and Synapse workspace.

Ingest on-premise data with Azure Data Factory using self-hosted integration runtime and a copy pipeline to stage tables in Azure Data Lake bronze layer as parquet.

Demonstrates building an Azure Data Factory pipeline to copy all on-premises SQL Server tables into the Data Lake bronze layer as parquet files using lookup and for-each.

Launch an Azure Databricks workspace, create a data transformation cluster, and mount a data lake using credential passthrough to process bronze, silver, and gold containers.

Orchestrate level-one and level-two data transformations from bronze to silver and silver to gold using Databricks, PySpark, and Delta format, including date conversions and naming convention standardization.

Discover how to build an end-to-end Azure Data Factory pipeline that runs Databricks notebooks for bronze to silver and silver to gold transformations, with real-time monitoring and Delta Lake storage.

Load gold-layer data from Azure data lake into Azure Synapse Analytics using serverless SQL, create views from delta tables, and automate via pipelines and stored procedures.

Connect Azure Synapse Analytics serverless SQL to Power BI to import all views from the gold_db, creating an interactive dashboard with modeled relationships and cross-filtered insights.

Learn to implement security and governance in a real-time data engineering project by creating an Azure Active Directory security group to grant resource group access using role assignments.

Test end-to-end data pipelines by inserting a row into on-prem SQL, copying to bronze via Azure Data Factory, transforming to silver and gold, and loading into Synapse for Power BI.