Azure Data Factory Project with Hands-On Lab for Beginners

Name: Azure Data Factory Project with Hands-On Lab for Beginners
Rating: 4.3 (37 reviews)

Create Azure Data Factory pipeline for absolute beginners

Created byTechLab Guru

Last updated 7/2024

English

What you'll learn

Introduction to Azure Data Factory
Setting Up Azure Data Factory
Creating and Managing Pipelines
Data Integration Technique

Course content

4 sections • 17 lectures • 1h 5m total length

Who are Data Engineers2:10
Data engineers integrate, transform, and consolidate data from various sources, handling structured, semi-structured, and unstructured data with Python, SQL, and Java, loading into data warehouses or lakes.
OLTP vs OLAP2:39
What is Azure Data Factory and its components5:39

What is PowerShell6:44
Master the basics of PowerShell, a scripting language for automating tasks in Azure Data Factory. Learn about variables, objects, data types (integer, string, double), and get type using PowerShell ISE.
How to Access characters of a string1:53
Learn how to access characters in a string using zero-based indices and square brackets, and perform string concatenation to create substrings.
Demo – Indexing and substring method4:15
Demo shows indexing strings in PowerShell ISE with square brackets and substring to pull a specified number of characters from a start index, relevant to Azure Data Factory tasks.
Conditional statement in PowerShell2:55
Demonstrates how conditional statements in PowerShell work using if, else, and elseif to compare two numbers in PowerShell ISE and determine which is greater.
foreach loop in PowerShell4:12
Master the foreach loop in PowerShell by reading server names from a text file, filtering Miami servers (first character M), and translating this logic to Azure Data Factory concepts.

Project Overview2:46
Create Azure Data Factory resource3:51
Demonstrates creating an Azure Data Factory resource in the portal and configuring a self-hosted integration runtime for on-premises data, plus exploring the studio for authoring pipelines.
Demo - Integration Runtime1:43
Install the integration runtime on an on premises file server using express setup, then verify the runtime shows running to confirm connectivity with a test virtual machine.
Demo - Linked Service5:43
Create a link service for the on-premises file system using the self-hosted integration runtime, test credentials, disable local folder path validation, then connect to Azure Data Lake Storage.
Demo - Data set2:41
Create on-premises and data lake storage datasets in Azure Data Factory, connect via integration runtime and link services to reference CSV sources and the destination container.
Demo - Get Metadata Activity5:11
Explore configuring the get metadata activity in a data factory pipeline to extract file names and child items from an on-prem data set for use with a for each loop.
Demo - Foreach Activity9:36
Demonstrates using a for each loop in azure data factory to process each file from get metadata, iterating child items and comparing dates via substring to utc formatted as yyyyMMdd.
Demo - Copy Activity3:23
Copy activity transfers only the csv file whose name matches today's date from the on-premises source to the data lake destination, using for each loop and item.name.

Requirements

No experience needed

Description

Welcome to "Azure Data Factory Project with Hands-On Lab for Beginners"! This course is designed to provide you with a comprehensive understanding of Azure Data Factory, Microsoft's cloud-based data integration service. Whether you're new to data engineering or looking to enhance your skills, this course will guide you through the essential concepts and practical applications of Azure Data Factory.

What You'll Learn:

Introduction to Azure Data Factory: Understand the fundamentals of Azure Data Factory, including its architecture and core components.
Creating and Managing Pipelines: Learn how to design, create, and manage data pipelines to automate data movement and transformation.
Data Integration: Explore various data integration patterns and techniques to connect and process data from different sources.
Hands-On Lab: Apply your knowledge through practical exercises and real-world scenarios in our hands-on lab environment.

Who Should Enroll:

Beginners: Individuals new to data engineering and cloud-based data integration.
Data Professionals: Data analysts, data engineers, and database administrators looking to expand their skillset.
IT Professionals: Those working in IT or related fields who want to learn about cloud-based data integration solutions.

By the end of this course, you'll have the skills and confidence to design, implement, and manage data pipelines using Azure Data Factory. Join us and take the first step towards mastering data integration in the cloud!

Who this course is for:

Beginners: Individuals new to data engineering and cloud-based data integration.
Data Professionals: Data analysts, data engineers, and database administrators looking to expand their skillset.
IT Professionals: Those working in IT or related fields who want to learn about cloud-based data integration solutions.

Azure Data Factory Project with Hands-On Lab for Beginners

What you'll learn

Explore related topics

Course content

Introduction3 lectures • 10min

Basics of Programing5 lectures • 20min

Azure Data Factory Project8 lectures • 35min

Important Links1 lecture • 1min

Requirements

Description

Who this course is for: