DP-203 Data Engineer Associate Certification Preparation

Microsoft Azure (formerly Windows Azure) is a cloud computing service created by Microsoft for building, testing, deploying, and managing applications and services through Microsoft-managed data centers.

Azure Provides Three services:

1. software as a service (SaaS),

2. platform as a service (PaaS).

3. infrastructure as a service (IaaS)

Azure supports many different programming languages, tools, and frameworks, including both Microsoft-specific and third-party software and systems. Azure was announced in October 2008, started with the codename "Project Red Dog", and released on February 1, 2010, as "Windows Azure" before being renamed "Microsoft Azure" on March 25, 2014.

In this course, you will learn :

1. How To Use The Azure Data Factory.

2. How To Use The Azure SQL Database.

3. How To Use Azure Blob Storage.

4. How To Use Azure Data Lake.

5. How To Use Azure DataBricks.

6. How To Use Different Azure Data Services In Different applications.

Exam DP-200: Implementing an Azure Data Solution:

Candidates for this exam must be able to implement data solutions that use the following Azure services:

• Azure Cosmos DB.

• Azure SQL Database.

• Azure Synapse Analytics (formerly Azure SQL DW).

• Azure Data Lake Storage.

• Azure Data Factory.

• Azure Stream Analytics.

• Azure Databricks.

• Azure Blob storage.

Topics We Cover In This Course:

• Azure SQL Database.

• Azure Cosmos DB.

• Azure Data Lake Storage.

• Azure Data Factory.

• Azure Databricks.

• Azure Blob storage.

• Azure Synapse Analytics (formerly Azure SQL DW).

Upcoming Modules;

• Azure SQL failover groups

• Azure Data Lake Analytics

• Introduction To Power BI

• HDinsight.

Candidates for this exam should have subject matter expertise integrating, transforming, and consolidating data from various structured and unstructured data systems into a structure that is suitable for building analytics solutions.

Azure data engineers help stakeholders understand the data through exploration, and they build and maintain secure and compliant data processing pipelines by using different tools and techniques. These professionals use various Azure data services and languages to store and produce cleansed and enhanced datasets for analysis.

This course covers, how to provisioning data storage services like Azure SQL, Storage account, Data lakes. In the Azure Data factory section, we cover how to transform your data, identifying performance bottlenecks, and accessing external data sources including on-premise SQL server and file systems.

Azure Data Factory (ADF):

The Azure Data Factory service is a fully managed service for composing data storage, processing, and movement services into streamlined, scalable, and reliable data production pipelines. The Azure Data Factory (ADF) is a service designed to allow developers to integrate disparate data sources. ADF or Azure Data Factory is a platform somewhat like SSIS or Alteryx in the Azure environment to manage the data you have both on-prem and in the cloud.

It provides access to on-premises data with the help of a software. By using this link we could connect to the on-premise file system as well as to on-premise SQL databases. From Azure Data Factory, you could access almost all azure services without any difficulties.

Access to on-premises data is provided through a data management gateway that connects to on-premises SQL Server databases and we will show you how to install this software and how to connect your on-premises environment with Azure cloud.

If you ever created any data transfer activities in Azure, or in SSIS, you will find it a similar tool. If you use ADF, you could focus on your data—the serverless integration service does the rest.

Topics In Azure Data Factory:

• Append Variable Activity

• Execute Pipeline activity

• ForEach activity

• Get Metadata activity

• If Condition activity

• Lookup activity

• Set variable activity

• Until activity

• Validation activity

• Data Flow activity

• Mapping data flow

  • Aggregate transformation.

  • Alter row transformation.

  • Conditional split transformation.

  • Derived column transformation.

  • Exists transformation.

  • Join transformation.

  • Lookup transformation.

  • The new branch mapping data flow transformation.

  • Select transformation.

  • Sink transformation.

  • Source transformation.

  • Azure Data Factory union transformation.

• Parameterizing

• Trigger In Azure Data Factory.

  • Manual Trigger.

  • Scheduled Trigger.

  • Tumbling window

  • Event Trigger.

    • Dynamic Data processing And Pipeline Execution Based on External Event.

• and many more (with real-life scenarios). Check out our course descriptions for updated information.

SQL Database -Cloud Database as a Service:

Azure SQL Database is a fully managed relational database with built-in intelligence supporting self-driving features such as performance tuning and threat alerts. According to Wiki, Microsoft Azure SQL Database is a managed cloud database provided as part of Microsoft Azure. A cloud database is a database that runs on a cloud computing platform, and access to it is provided as a service. Managed database services take care of scalability, backup, and high availability of the database.

Azure SQL Database: Azure SQL Database is a relational database-as-a-service (DBaaS) based on the latest stable version of Microsoft SQL Server. It is a fully managed Platform as a Service (PaaS) Database Engine that handles most of the database management functions such as upgrading, patching, backups, and monitoring without user involvement.

In this course, we will show you how to launch the Azure SQL database in five minutes, with and without sample data.  We will show you, how to use SQL elastic pools.

Elastic pools help you manage and scale multiple Azure SQL databases. According to Azure documentation, SQL Database elastic pools are a simple, cost-effective solution for managing and scaling multiple databases that have varying and unpredictable usage demands. With an elastic pool, you determine the amount of resources that the elastic pool requires to handle the workload of its databases, and the amount of resources for each pooled database.

Azure SQL Database is always running on the latest stable version of the SQL Server Database Engine and patched OS with 99.99% availability.

The databases in an elastic pool are on a single Azure SQL Database server and share a set number of resources at a set price. By the end of this course, you will have a clear idea about how to configure SQL elastic pool.

Geo-Replication:

Active geo-replication is an Azure SQL Database feature that allows you to create readable secondary databases of individual databases on a SQL Database server in the same or different data center (region). We will show you how you could configure a Geo-replication and force failover to the secondary database manually.

Azure Cosmos DB:

Azure Cosmos DB is Microsoft's globally distributed, multi-model database service. With a click of a button, Cosmos DB enables you to elastically and independently scale throughput and storage across any number of Azure regions worldwide. In our course, we will see.

• How you could create a cosmos DB account,

• How to Create Databases inside your cosmosDB account

• How to insert data into CosmsoDB containers.

• How to Restive data that you saved in cosmos DB tables or containers by using SQL

Introduction to Azure Storage:

Azure Storage is Microsoft's cloud storage solution for modern data storage scenarios. Azure Storage offers a massively scalable object store for data objects, a file system service for the cloud, a messaging store for reliable messaging, and a NoSQL store.

In this course, we will cover how to create a storage account, how to create containers and file systems and how to upload data into these services and how to access these storage services from different azure data solutions services like data factory, data bricks, and SQL databases.

Azure Data Lake Storage:

Azure Data Lake Storage, is a fully-managed, elastic, scalable, and secure file system that supports HDFS semantics and works with the Hadoop ecosystem. ‎Azure Data Lake Storage Gen2 is a set of capabilities dedicated to big data analytics, built on Azure Blob storage. Data Lake Storage Gen2 is the result of converging the capabilities of our two existing storage services, Azure Blob storage and Azure Data Lake Storage Gen1.

In this course we will see, how to create data lakes, How to move CSV data from azure blob storage to azure data lake using azure data factory. How to read your data (Azure Data lake) using azure Databricks.

Azure Databricks:

Azure Databricks provides the latest versions of Apache Spark and allows you to seamlessly integrate with open source libraries. Spin up clusters and build Bigdata applications. According to Databricks documents, Azure Databricks is a fast, easy, and collaborative Apache Spark™ based analytics platform optimized for Azure. Azure Databricks supports Python, Scala, R, Java, and SQL, as well as data science frameworks and libraries including TensorFlow, PyTorch, and scikit-learn.

In this course, we will show you how to configure Azure Data bricks, How to launch a cluster, how to create notebooks.

Azure Synapse Analytics (Azure SQL Data Warehouse):

Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. Azure Synapse is Azure SQL Data Warehouse evolved. In this course, you will learn how to create an Azure SQL pool, access a data lake storage account (how to use PolyBase external tables to load data from Azure Data Lake Storage). Will demonstrate how to create a master key and database scoped credential. How to create external tables and external data sources. Finally, we will see how to load data into Azure Data Warehouse from an external table by using the create table as a select command.

Demo 1:

1. Create a Pipeline in Azure Data Factory.

2. Create Input Connections to a source.

3. Create Input Data Set.

4. Create Output Connections to destinations.

5. Create an Output Data set.

6. Create A copy Activities to copy data from on-premise to Azure Blob storage.

7. Create A copy Activities to copy data from Blob to Azure SQL Database.

8. Create A copy Activities to copy data from on-premise to SQL Database.

9. Run Your Copy Activities and validate all the settings

Demo-2:

1. Migrate Data from On-premise SQL Server to Azure SQL database Without any external services.

Demo 3:

1. Create An Azure DataBricks.

2. Connect To Azure Data Lake.

3. Create a Cluster To Run our notebook

4. Configure Azure Databricks Data lake configurations.

5. Assign permission to your external Applications.

6. Read CSV data saved inside Azure Data lake using a Python notebook.

Demo 4:

1. Create Your First Data Flow In Azure Data Factory.

2. Configure The Source Data flow.

3. Learn To use Filter Conditions.

4. Learn To configure Sink (Destinations) in Azure.

5. Run your Azure Data flow and copy data from Azure Blob Data Store And Save filtered result In Azure Data lake.

Demo 5:

1. Access On-premise SQL Server.

2. Create Different  Data Set by Executing Custom Stored Procedure By Passing Dynamic Parameter

3. Save This Data Set Into Data Lake By Creating Custom Filename.

4. Trigger this action Azure Data Factory.

Demo 6:

1. Create dynamic result with help of custom parameters and in for each activity.

2. Save the result into Azure Data lake with a dynamic name.

Demo 7;

Run your activities N times with the help of Until loop (Do loop concepts of programming language)

Demo 8:

• Create an Azure Data warehouse Pool by using Azure Portal.

• Create an Azure Data warehouse by using SQL statements from SSMS.

• Connect and execute SQL statements against  Azure SQL Data warehouse.

• Execute SQL statements against Data in Azure Data lake using PolyBase external tables And load data from Azure Data Lake Storage into Azure Data warehouse.

• Learn to use external tables and external data sources in Azure SQL Data warehouse.