Learn Azure Data Engineering with 20+ Real Time Projects

Set up an adls account in Azure with hierarchical namespace, create a lake house with silver and gold folders, and use pyspark to clean data for a Power BI dashboard.

Create a new fabric workspace and lakehouse, set up a medallion architecture with branch, silver, and gold layers, and migrate ADLs CSV data to parquet via a data pipeline.

Create a data pipeline to read files from Azure Data Lake Gen2, get metadata for file names, and copy each file into the lakehouse as parquet.

Learn to clean messy data with PySpark, build silver and gold data layers in a lakehouse, and visualize insights with Power BI through end-to-end data engineering.

Build an end-to-end Azure data engineering project: ingest rest API data into ADLS with Azure Data Factory, process with PySpark in Synapse, and publish a daily purchases and revenue report.

Perform schema validation in azure data factory by comparing incoming csv structures against a predefined schema using get metadata and an if condition, ensuring only matching files are processed.

Execute an end-to-end retail analytics project on Databricks, applying medallion architecture to unify and clean CSV/JSON data with PySpark, then build a dashboard for revenue and average order value.

Create a bronze data layer in Databricks by provisioning a catalog, database, and volume, uploading raw files, and reading csv and json data to begin cleaning in a medallion architecture.

Clean and transform raw order and customer data with PySpark, standardizing IDs and dates, handling nulls, deduplicating, and storing silver delta tables for orders and customers.

Build a gold layer from silver tables by left joining on customer id, deriving year and month from order dates, and aggregating total sales for dashboards.

Transfer data from a REST API to Azure Data Lake Storage using Azure Data Factory, copying only ID, email, first name, last name, and author to a CSV sink.

Build an end-to-end retail data pipeline in Microsoft Fabric using medallion architecture (bronze, silver, gold) to clean, unify, and visualize data with PySpark and Power BI.

Create a medallion lakehouse in fabric and implement a data pipeline to copy data from adls to parquet in bronze, silver, and gold layers.

Read parquet data from the medallion lakehouse with PySpark, clean orders, returns, and inventory, build bronze, silver, and gold delta tables, and publish the gold data to Power BI.

Deliver an end-to-end Azure data engineering project for retail, using Azure Data Factory, Azure Data Lake Storage, Databricks, medallion bronze-silver-gold, and Power BI visualizations.

Execute an end-to-end Azure Data Factory project to implement SCD type 1, loading daily CSV data from blob storage into Azure SQL Database using upsert, update, and insert operations.

Build an end-to-end Azure Data Factory pipeline using data flow to implement SCD type 2, inserting new records and updating existing ones while tracking historical and current records.