Azure Data Engineer Technologies for Beginners [DP-200, 201]
4.5 (1,007 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
5,762 students enrolled

Azure Data Engineer Technologies for Beginners [DP-200, 201]

Microsoft Azure SQL Database, Data Lake, Data Factory, Synapse Analytics, Cosmos DB, Databricks,HDInsight,DP-200, DP-201
Bestseller
4.5 (1,007 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
5,762 students enrolled
Created by Eshant Garg
Last updated 7/2020
English
English [Auto]
Current price: $13.99 Original price: $19.99 Discount: 30% off
5 hours left at this price!
30-Day Money-Back Guarantee
This course includes
  • 31 hours on-demand video
  • 47 articles
  • 35 downloadable resources
  • 1 Practice Test
  • Full lifetime access
  • Access on mobile and TV
  • Assignments
  • Certificate of Completion
Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • You will be able to identify right Azure SQL Server deployment option, purchasing model and service tier according to requirements and will be able to deploy it in cloud.
  • You will able to deploy Azure Synapse Analytics (formerly known as Azure SQL Data warehouse) in Azure Cloud environment.
  • You will have good internal MPP architecture understanding, and so you will be able to analyse your on-premises data warehouse, and migrate data to Azure Data Warehouse.
  • You will be able to create Azure Data Lake Gen1 storage account, populate it will data and analyze it using U-SQL Language.
  • You will understand Azure Data Factory key components and advantages. You will be able to create, schedule and monitor simple pipelines.
  • We will understand how HDInsight makes Hadoop easy, and we will go through simple demo where we’ll fetch data from Data Lake, process it through Hive and later will store data in SQL Server.
Requirements
  • Basic Database concepts
  • Azure free subscription
Description

Why Microsoft Azure Data Engineer?

  • Microsoft Azure Date Engineering is one of the fastest-growing and in-demand occupations among Data Science practitioners.

  • According to a 2019 Dice report, there was an 88% year-over-year growth in job postings for data engineers, which was the highest growth rate among all technology jobs.

  • If you are interested in this domain, there cannot be a better time to start learning these Microsoft Azure Data Solution technologies.


Expected Outcomes

After this course:

  • Microsoft Azure SQL Server - You will be able to identify the right Azure SQL Server deployment option, purchasing model and service tier according to requirements and will be able to deploy it in the cloud. You will be able to set up a security configuration to secure your database.

  • Microsoft Azure SQL Data Warehouse - You will able to deploy Azure Synapse Analytics (formerly known as Azure SQL Data warehouse) in Azure Cloud environment. You will have good internal MPP architecture understanding, and so you will be able to analyze your on-premises data warehouse and migrate data to Azure Data Warehouse.

  • Microsoft Azure Data Lake - You will be able to create Azure Data Lake storage account, populate it will data using different tools and analyze it using Databricks and HDInsight

  • Microsoft Azure Data Factory - You will understand Azure Data Factory's key components and advantages. You will be able to create, schedule and monitor simple pipelines.

  • Hadoop Basics - You will learn the fundamental understanding of the Hadoop Ecosystem and 3 main building blocks. This module will prepare you to start learning Big Data in Azure Cloud using HDInsight.

  • Microsoft Azure HDInsight - We will understand how HDInsight makes Hadoop easy, and we will go through simple demo where we’ll fetch data from Data Lake, process it through Hive and later will store data in SQL Server.

  • Cosmos DB - You will understand all basic concepts of Cosmos DB, and will be able to create a new database, container, documents, and choose right partition key, configure the global distribution and use other important features

  • Streaming Service - I will add details soon

  • Databricks - I will add details soon

  • Storage Service - I will add details soon

Intended Audience

  • Beginners in Microsoft Azure Platform

  • Microsoft Azure Data Engineers

  • Microsoft Azure Data Scientist

  • Database and BI developers

  • Database Administrators

  • Data Analyst or similar profiles

  • On-Premises Database related profiles who want to learn how to implement these technologies in Azure Cloud.

  • Anyone who is looking forward to starting his career as an Azure Data Engineer.

Level

  • Beginners (100)

  • "if you are already experienced and working on these technologies, this may not be the best course for you."

  • I have a few crash courses (Free) for absolute beginners, you can find links on my website.


Prerequisites

  • Basic T-SQL and Database concepts

  • Microsoft Azure Free trial Subscription

Language

  • English

  • If you are not comfortable in English, please do not take a course, captions are not good enough to understand the course.

What's inside

  • Video lectures, PPTs, Demo Resources, Quiz, Assignment, other important links

  • Full lifetime access with all future updates

  • Certificate of course completion

  • 30-Day Money-Back Guarantee

Course updates

I am consistently adding new content in terms of both lectures and demos. Please find below logs for past and upcoming changes.

Coming soon

  • July 2020 - Monitoring and Optimization of services

Previous Updates:

  • July 2020 - AMS (Azure Monitoring Service)

  • July 2020 - Re-recorded Azure Data Factory with all new updates

  • July 2020 - NoSQL Databases in Microsoft Azure (Azure Blob Storage)

  • July 2020 - Added Azure Databricks Module

  • July 2020 - Added Azure Streaming Analytics Module

  • June 2020 - Added Azure Cosmos DB Module

  • May 2020 - Full-length course on Data Lake

  • Apr 2020 - Added modules on Azure cloud computing basics, and Data Engineer profile

  • Apr 2020 - Added crash course on cloud computing and data warehouse

  • Mar 2020 - Added Azure Synapse Analytics

  • Feb 2020 -  Added Azure SQL Database

  • Oct 2019 -  Added Azure HDInsight

  • Sep 2019 -  Added Azure Data Factory

  • Aug 2019 - Course Created on Azure Data Lake


Course In Detail

By the end of this course

  • Microsoft Azure SQL Server -

    • You will know the advantages of Azure Database over on-premises Database

    • You will be able to provision all three types of Azure Database PaaS Deployments (Single, Elastic Pool, Managed Instance)

    • You will learn different purchasing models and Service tiers and how to configure them according to requirements.

    • You will be able to provision Azure Database in Virtual Machine (IaaS)

    • You will be able to deploy Elastic Pool Database and able to add/remove more databases from the pool.

    • You will be able to deploy the Managed Instance Database, configure it and connect it with your SQL Server Management studio.

  • Microsoft Azure Synapse Analytics Service

    • You will learn why we should consider warehousing solutions in the cloud.

    • You will learn about Microsoft's brand new Azure Synapse analytics service, and how this service brings together enterprise data warehousing and Big Data analytics, and provide a unified experience to ingest, prepare, manage, and serve data for immediate BI and machine learning needs.

    • You will learn the difference between Traditional vs Modern vs Synapse Architecture.

    • You will learn Azure famous MPP (Massive Parallel Processing) architecture

    • You will also learn many internal concepts like Data Distribution, Sharding, and partitioning.

    • You will learn different migration methods and best practices.

    • You will learn PolyBase setup (Best Loading method)

    • You will perform a lot of demos to migrate data to Azure SQL Data warehouse using SSIS, Data Factory and PolyBase.

  • Microsoft Azure Data Lake Gen1

    • You will learn the limitations of traditional database systems to handle the Big Data revolution.

    • You will learn the difference between Azure Data Lake, SSIS, Hadoop and Data Warehouse.

    • You will be able to create Azure Data Lake Gen1 storage account, populate it will data and analyze it using U-SQL Language.

  • Microsoft Azure Data Factory -

    • You will understand Azure Data Factory's key components and advantages.

    • You will be able to create, schedule and monitor simple pipelines.

  • Hadoop Basics

    • You will learn a fundamental understanding of the Hadoop Ecosystem and 3 main building blocks.

    • This module will prepare you to start learning Big Data in Azure Cloud using HDInsight.

  • Microsoft HDInsight -

    • You will learn what are the challenges with Hadoop and how HDInsight solves these challenges.

    • You will also learn Cluster types, HDInsight Architecture and other important aspects of Azure HDInsight.

    • You will also go through demo where we’ll fetch data from Data Lake, process it through Hive and later will store data in SQL Server.

Some students Feedback

  • One of the most amazing courses i have ever taken on Udemy. Please don't hesitate to take this course. The instructor is really professional and has a great experience about the subject of the course. - Khadija Badary

  • Very nicely explained most of the concepts. a must have course for beginners - Manoranjan Swain

  • I appreciate this course explaining everything in great detail for a beginner. This will assist me in overcoming challenges at my work - Benjamin Curtis

  • Good course for Beginners. Labs are really helpful to grasp the concept. Thank you - Sapna

Who this course is for:
  • Beginners in Azure Platform
  • Database Developer
  • Database Administrators (DBA)
  • Business Intelligence (BI) Developers
  • Data Engineers
  • Data Scientist
  • Data Analyst or similar profiles
  • Other on-premises Database related profiles who want to learn how to implement these technologies in Azure Cloud.
  • Anyone who is looking forward to start his career as a Azure Data Engineer
Course content
Expand all 311 lectures 30:57:50
+ Course Overview and Set Expectations
6 lectures 07:32
Before you start...
01:29
QAs related to course - Imp Note
00:34
DP-200 study material
01:08
DP-201 study material
01:05
+ Introduction to Azure Cloud Computing
14 lectures 01:30:10
Download PPT of this Module
00:01
Create Azure free subscription
06:13
Azure Services Overview
11:08
Managed and Unmanaged Service
05:00
Resource Management group and Subscription
08:34
Resource Groups
08:16
Resource Provider
02:29
Tagging
03:09
Access Control
05:33
Demo - Create Azure SQL Database service
15:21
Delete resources and set budget
06:59
Conclusion
01:29
+ Introduction to Data Engineer Profile
6 lectures 13:10
Download PPT of this Module
00:01
Introduction
00:38
How did Data Engineer profile evolve?
02:35
Introduction to Data Engineer technologies
03:45
Conclusion
00:40
+ ****** Azure SQL Database ******
42 lectures 03:01:23
Module Introduction
04:56
Download PPT - Azure SQL Database
00:02
**** SECTION **** - Introduction to Azure SQL Database
00:01
Introduction
01:08
Why choosing SQL Server in Azure
02:58
SQL Server PaaS deployment options
03:50
Further References from Microsoft Docs
00:03
Practice Test
4 questions
**** SECTION **** - Azure SQL Server in Virtual Machine (IaaS)
00:02
Introduction
00:31
SQL Server in Azure Virtual Machine
01:57
Further References from Microsoft Docs
00:02
Practice Test
2 questions
**** SECTION **** - Azure Single Database
00:01
Introduction
01:08
Demo - Azure Single Database
15:37
Purchasing models and Service Tier
13:23
Azure Database vs Azure Data warehouse
02:33
Further References from Microsoft Docs
00:02
Practice Test
4 questions
**** SECTION **** - Elastic Database Pool
00:01
Introduction
00:39
Azure Elastic Database
04:16
Demo - Azure Elastic Database
14:01
Further References from Microsoft Docs
00:02
Practice Test
2 questions
**** SECTION **** - Managed Instance Database
00:01
Introduction
01:00
Azure Managed Instance Database
04:17
Difference between on-premises and Managed Instance
01:53
Migration options for Managed Instance
01:18
Service tiers for Managed Instance
01:44
Management operations
01:45
Demo - Managed Instance
17:42
Further References from Microsoft Docs
00:02
Practice Test
2 questions
**** SECTION **** - Azure Database Security
00:01
Introduction
01:01
Azure Database Security options
08:11
Azure Managed Instance advance security
01:25
Further References from Microsoft Docs
00:02
Practice Test
3 questions
Monitoring through portal
15:45
Monitoring through external tools
09:15
Intelligent performance and Automatic Tuning
14:16
Query Performance Insight
13:00
Azure SQL Database Practice Test
7 questions
+ ****** Azure SQL Data Warehouse *****
57 lectures 05:08:13
Data Warehouse - crash course
00:09
Module Introduction
03:12
Azure Synapse - PPT and Demo Resources
00:02
**** SECTION **** - Introduction to Azure Synapse Analytics Service
00:02
Introduction
02:45
Why warehousing in cloud
02:48
Traditional vs Modern Warehouse architecture
14:24
Azure Synapse a Game changer
06:48
Azure Synapse Benefits
03:00
Modern vs Azure Synapse Architecture
03:20
Demo Azure Synapse Analytics
16:10
Azure Synapse Studio
25:13
Synpase unified experience
02:20
Summary
01:53
Practice Test
3 questions
**** SECTION **** - Azure Synapse Internal and Architecture
00:01
Introduction
01:47
Azure Synapse MPP Architecture
07:10
Storage and Sharding patterns
05:29
Data Distribution and Distributing Keys
06:05
Data Types and Table Types
05:20
Partitioning
04:48
Best Practices for Fact and Dimension tables
02:49
Demo - Analyse data distribution before migration to azure
12:19
Summary
02:29
Practice Test
9 questions
**** SECTION **** - Data Migration to Azure Synapse Data Warehouse
00:02
Introduction
01:02
5 best practices for Data Load
05:13
DIfferent loading methods
02:38
Loading with SSIS vs PolyBase
03:35
Demo - Loading with SSIS
05:25
Demo - Loading with Polybase
27:10
Demo - Loading with Data Factory
16:43
Summary
00:55
Practice Test
8 questions
Assignment Question - Lab: Lake to Synapse
02:51
Assignment Solution - Lab: Lake to Synapse
22:46
**** SECTION **** - Security layers in Azure Synapse Service
00:02
Introduction
01:29
Advance Data Security
07:03
Auditing
04:03
Network Security
03:09
Transparent data encryption
01:38
Dynamic Data Masking
04:05
Access Management
05:30
Summary
01:47
Practice Test
7 questions
**** SECTION **** - Configuring and Optimizing Azure Synapse Service
00:02
Introduction
01:37
Configuration Options
08:01
Performing other tasks
03:56
Backup and Restore
07:57
Price Optimization
05:32
Managing Workloads
10:05
Monitoring
12:15
Support and Troubleshooting
03:21
Delete Resources
03:52
Summary
01:16
Practice Test
6 questions
DP-200 Quick Notes/cheat sheet
01:12
+ ****** Azure Data Lake ******
55 lectures 04:50:01
Data Lake - PPT and Demo Resources
00:01
Data Lake vs Hadoop
01:34
How Data Lake Gen2 evolved?
05:17
Blob Storage vs Data Lake
04:37
Hierarchical namespace
06:12
Demo - Create Azure Data Lake Gen2 Account
11:01
Demo - Data Lake Gen2 account overview
10:16
Data Lake Gen2 other features
06:00
**** SECTION **** - Data Lake Ingestion through diff tools
00:02
Explore different tools to Ingest Data in Azure Data Lake Gen2
11:03
Demo - On-Premises to Data Lake Gen2 using portal and storage explorer
09:11
Demo - On-Premises to Data Lake Gen2 using azcopy
06:04
Demo - Azure Blob to Data Lake Gen 2 using Data Factory
09:04
Demo - Azure SQL Database to Data Lake Gen2 using Data Factory
14:13
Demo - Amazon S3 to Data Lake Gen2 using Data Factory
08:16
**** SECTION **** - Data flow around Data Lake
00:01
Data flow around Data Lake
04:52
Data Lake and Transient clusters
02:13
Data Processing using HDInsight
00:07
**** SECTION **** - Data Lake Processing through Databricks
00:01
Imp Note about this Demo
00:50
Demo overview
04:35
Provision Databricks, Clusters and workbook
11:42
Mount Data Lake to Databricks DBFS
12:28
Explore, Analyze, Clean, Transform
18:04
**** SECTION **** - Data Lake Security Layers
00:01
Introduction
02:01
Storage Access Keys
03:24
SAS - Shared Access Signature
07:16
Azure Active Directory
04:44
Role Based Access Control (RBAC)
05:33
Access Control list (ACL)
06:23
Firewalls and Virtual Networks
03:53
Encryption in Transit
03:31
Encryption at Rest
02:36
Advanced threat protection
02:06
**** SECTION **** - Data Lake Monitoring and Optimization
00:01
Activity Log
04:41
Demo: Activity Logs
06:42
Metrics
03:38
Demo: Metrics
08:47
Demo: Insights
08:26
Demo: Alerts
08:25
Diagnostic settings
08:16
Demo: Diagnostic settings
08:20
Optimization
05:39
**** SECTION **** - Demo - Azure Data Lake Gen1
00:01
How to upload data in Data Lake Gen1
03:11
Write U-SQL Script to transform data
05:57
Schedule a job in Data Lake
03:12
Summary
00:24
+ ****** NoSQL Databases in Microsoft Azure ******
10 lectures 01:54:40
Introduction to NoSQL
09:23
SQL vs NoSQL
04:21
4 types of NoSQL Databases
09:24
NoSQL Offerings by Microsoft Azure
09:06
Azure Storage Introduction
06:28
Demo: Part 1 Provision Azure Storage Account
17:44
Demo Part 2: Azure Storage Replication
12:57
Demo: Part 3 Continue Provisioning Azure Storage account
20:33
Azure Blob Storage
10:50
Managing the Data LifeCycle
13:54
+ ****** Azure Cosmos DB ******
26 lectures 03:19:53
Cosmos DB - PPT and Demo Resources
00:02
Problem Statement How Cosmos DB Evolved
02:38
Cosmos DB Features
11:24
Cosmos DB - Multi Model 5 APIs
12:21
Azure Table Storage vs Cosmos DB Table API
04:36
Provision Cosmos DB Account
10:34
Cosmos DB - Database , Containers and items
04:27
Cosmos DB - Throughput and Request units
09:18
Cosmos DB - Horizontally Scalable
02:49
Cosmos DB - Partition and Partitioning key
06:10
Cosmos DB - Dedicated vs Shared throughput
06:36
Cosmos DB - Avoiding hot partitions
05:24
Cosmos DB - Single partition vs Cross partition
03:32
Cosmos DB - Composite Key
03:14
Cosmos DB - Partition key best practice
07:48
Cosmos DB - Automatic Indexing
05:20
Demo Insert and query data in your cosmos DB
12:41
Cosmos DB - Time to Live feature
05:48
Cosmos DB - Globally Distribution feature
13:23
Cosmos DB - Multi Master feature
08:38
Cosmos DB - Manual vs Automatics Failover
07:36
Cosmos DB - 5 consistent levels
14:16
Cosmos DB - Azure CLI
07:02
Cosmos DB - Pricing
06:50
Cosmos DB - Monitoring through AMS (Azure Monitor Service)
14:43
Cosmos DB - Monitoring through Cosmos DB Portal
12:43
+ ****** Azure Streaming Analytics ******
12 lectures 01:33:20
What is Live Event Processing?
09:19
Introduction to Azure Streaming Analytics Service
06:18
Streaming Analytics - What is time windowing
02:22
Streaming Analytics - Tumbling Window
02:33
Streaming Analytics - Hopping Window
02:13
PPT and Demo Resources
00:01
Streaming Analytics - Sliding Window
02:22
Streaming Analytics - Session Window
06:06
Streaming Analytics - Processing Blob Storage Input
18:43
Streaming Analytics - Processing IOT Hub Input
18:49
Stream Analytics - Monitoring
14:12
Stream Analytics - Optimization
10:22
+ ****** Azure Databricks ******
11 lectures 01:52:37
PPT and Demo Resources
00:02
Spark Basics
27:25
Why Spark is difficult? Why Databricks Evolved?
05:08
Why Databricks in Cloud? Introduction to Azure Databricks
04:31
Demo overview
04:32
Demo: Provision Databricks, Clusters and workbook
11:22
Demo: Mount Data Lake to Databricks DBFS
12:24
Demo :Explore, Analyze, Clean, Transform and Load Data in Databricks
18:04
Azure Databricks Clusters
12:56
Azure Databricks other Important Components
09:34
Databricks - Monitoring
06:39