Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Build End - End Azure Data Engineering Project

Name: Build End - End Azure Data Engineering Project
Rating: 4.1 (20 reviews)

Build Your Azure Data Engineering Skills with Hands-On Experience in an End-to-End Project - Develop Real-World Solution

Created bySagar Prajapati

Last updated 6/2024

English

English [Auto],

What you'll learn

In this course, we will teach you how to build an end-to-end data engineering project on Azure.
You'll learn to gather data from various sources, store it in Azure Data Lake Storage, process it with Azure Data Factory, and apply analytics to the processed
This course is suitable for anyone with knowledge of SQL and Python. Whether you're a software engineer, data analyst, or IT professional
You'll gain valuable insights into building scalable and reliable data pipelines on Azure

Course content

13 sections • 41 lectures • 15h 0m total length

Architecture14:00
Discover delta lake architecture with bronze, silver, and gold layers that ingest raw data, refine it with cleansing, apply business logic, and enable time travel via delta log.
Introduction of ADF13:58
Learn to set up an Azure Data Factory, a cloud ETL service, and explore pipelines, data sources, link services, triggers, datasets, integration runtimes, and monitoring.
Introduction of Blob Storage14:09
Learn how Azure blob storage stores unstructured data in containers and how to create a blob storage account in the portal, including redundancy and hot or cool access tiers.
Introduction of AzureDataLake Gen211:01
Explore Azure Data Lake Gen2 as a centralized analytics data lake built on Azure Blob storage, enabling raw, cleanse, and publish data layers. Understand hierarchical namespace and directory-level access controls.
Introduction of Key Vault6:54
learn how to create and configure Azure Key Vault, a cloud service for securely storing and accessing secrets, including setting access policies, RBAC, and managing secrets, keys, and certificates.
What is SPN and How to register SPN?7:04
Create a service principal (SPN) via Azure app registration, generate a client secret or certificate, and manage separate dev and prod SPNs with app IDs and tenant IDs.
Introduction of SQL9:33
Explore SQL databases and SQL data warehouse concepts, and learn to create Azure SQL database as a source, considering cost and authentication, with query editor access to tables, stored procedures.
Introduction of Azure Databricks8:02
Explore how to create an Azure Databricks workspace in the portal, and learn Apache Spark-based processing with SQL, PySpark, and Scala, plus data science and engineering workflows.
Introduction of Azure Logic App6:36
Discover how Azure Logic Apps automate workflows using connectors and triggers, create and deploy Logic Apps in the Azure portal, and integrate with ADF pipelines and emails.
Introduction of Azure Automation Account4:13
Explore Azure automation account, a cloud-based automation platform with hybrid worker support for on-premise systems, pay-as-you-go pricing, and 500 free minutes of execution for runbooks written in PowerShell or Python.
What is API? and How to Register API_Key ?4:19
Learn how to create an API account, register and copy your API key, and access the airline endpoint to fetch data.

Create LinkedServices and Grant Access to Resource22:19
Discover how to use Azure Data Factory to copy data from Gen2 storage, SQL, and API into Azure Data Lake with date-partitioned folders, linked services, and access via Key Vault.
Create Datasets and Ingest CSV files31:19
Create datasets in Azure Data Factory, ingest CSV files from data lake Gen2, and copy to a raw container using dynamic pipelines.
Copy SQL tables from Server to DataLake Gen2 using parameterization20:00
Copy sql tables from sql databases to Azure Data Lake Gen2 in parquet format using parameterization, with dynamic datasets and linked services.
Unzip files and Ingest them into Data Lake Gen2 with single pipeline21:08
Unzip the zip file and ingest the 2005–2008 csv files into a delta lake gen2 with a single azure data factory pipeline.
Fetch Data from API using Rest API and Web Activity to fetch credentials19:24
Learn how to fetch data from a REST API using Azure Data Factory and Web Activity, securely handling credentials via Key Vault and copying into a raw data lake.
Setup Repository and commit ADF changes to Github Repository7:51
Learn how to save ADF code to a GitHub repository, configure develop and ADF publish branches, and commit ADF changes.
Updates on ADF Pipeline0:59
Update your ADF pipeline by configuring the copy data lake activity, checking compression settings (zip deflate or none), and saving to prevent raw data in the data lake.
Create Mount Point of Azure Blob Storage using Databricks14:09
Learn how to link Azure Databricks with a key vault, create a mount point to Azure Blob Storage for a raw data lake, and manage access with SAS tokens.
Create Mount Point of Azure DataLake GEN2 and Dumping PDF files using Databricks22:05
Create mount points for Azure Data Lake Gen2, read PDFs with Tabula in Databricks, and dump them into a raw data lake with date-based folders.
Copy PDF files from Azure Blob Storage using parameterization to Azure Datalake16:33
Parameterize a notebook to read PDF files from an Azure blob container and dump them into a data lake, filtering PDFs and handling dynamic file names and pages.

Clean the Data using AUTOLOADER in Databricks - Part 124:36
Learn to clean data with Databricks autoloader, create a proper schema, and store cleansed data in delta format, using schema and file checkpoints to load only latest files.
Clean the Data using AUTOLOADER in Databricks - Part 221:47
Cleanses multiple tables in a Databricks pipeline, loading delta format data, validating schemas, and creating cleansed delta tables with SQL and Spark.
Clean the Data using AUTOLOADER in Databricks - Part 37:47
Cleanse raw airport data in Databricks using autoloader, split description into city, country, and airport columns, and standardize naming with a metadata-driven JSON configuration to load into Delta Lake.
Clean the Data using Batch processing in Databricks18:38
Learn batch data cleansing in Databricks by building a cleansing layer for airlines data, using explode to flatten, selecting fields, and saving as delta with overwrite, plus parameterizing notebooks.
Data Quality checks on Cleansed Layer using Delta42:12
Build data quality checks for a delta table by comparing daily counts with the previous version, using delta history and spark sql, and generate alerts when deviations exceed a threshold.
Commit Databricks Notebook using Repo feature in GitHub11:26
Learn to commit Databricks notebooks to a GitHub repository using the repo feature, connect Databricks workspace to GitHub, configure tokens, create branches, and push changes.

Multiple Ways to Send Data files to End Users37:51
Discover end-to-end Azure data delivery workflows using Logic Apps, ADF, and Databricks to send CSV or Excel attachments by email to end users, with SQL and Python options.
Optimize way to send Data Files to End Users14:16
Optimize the end-to-end data delivery by replacing extra sql steps with an Excel lookup and a logic app call, then use for each and concurrency controls for parallel email sends.

Requirements

Basic Knowledge of Azure, Databricks

Description

In this course, we will teach you how to build an end-to-end data engineering project on Azure. You'll learn to gather data from various sources, store it in Azure Data Lake Storage, process it with Azure Data Factory, and apply analytics to the processed data using Azure Databricks. Throughout the course, you'll build a project from scratch, following the best practices and design patterns used by professional data engineers.

By the end of the course, you'll have a deep understanding of the Azure data engineering stack and be able to apply your skills to real-world projects. This course is suitable for anyone with knowledge of SQL and Python. Whether you're a software engineer, data analyst, or IT professional, you'll gain valuable insights into building scalable and reliable data pipelines on Azure. Start your journey to becoming an Azure data engineering professional with this comprehensive course.

Enroll in this course now and clear your Azure Interviews with 100% guaranteed. I have used below skills

1. ADF

2. Databricks

3. ADLS Gen2

4. Blob Storage

5. Service Principle

6. IAM

7. Azure Synapse

8. Rest API

9. Python

10. SQL

11. Github

12. Azure Devops

Build Your Azure Data Engineering Skills with Hands-On Experience in an End-to-End Project - Develop Real-World Solutions and Earn In-Demand Skills Today!

Who this course is for:

Students who are looking to change the job

Build End - End Azure Data Engineering Project

What you'll learn

Explore related topics

Course content

Let's understand Azure Resources and How to create them11 lectures • 1hr 40min

Data Preparation and Uploading into the respective sources1 lecture • 11min

Ingest Data from multiple source10 lectures • 2hr 56min

Cleaning Process and Data Quality Check6 lectures • 2hr 6min

Create Dimesions and Fact tables1 lecture • 55min

Publish the data into SQL DataBase using ADF1 lecture • 52min

Setup complete ADF Pipelines1 lecture • 44min

Send Data to End user using Logic App2 lectures • 52min

Run Automation Account script using Runbook1 lecture • 1hr 3min

Data Quality Layer between Mart and Publish1 lecture • 29min

Requirements

Description

Who this course is for: