Udemy
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
Development
Web Development Data Science Mobile Development Programming Languages Game Development Database Design & Development Software Testing Software Engineering Software Development Tools No-Code Development
Business
Entrepreneurship Communication Management Sales Business Strategy Operations Project Management Business Law Business Analytics & Intelligence Human Resources Industry E-Commerce Media Real Estate Other Business
Finance & Accounting
Accounting & Bookkeeping Compliance Cryptocurrency & Blockchain Economics Finance Finance Cert & Exam Prep Financial Modeling & Analysis Investing & Trading Money Management Tools Taxes Other Finance & Accounting
IT & Software
IT Certifications Network & Security Hardware Operating Systems & Servers Other IT & Software
Office Productivity
Microsoft Apple Google SAP Oracle Other Office Productivity
Personal Development
Personal Transformation Personal Productivity Leadership Career Development Parenting & Relationships Happiness Esoteric Practices Religion & Spirituality Personal Brand Building Creativity Influence Self Esteem & Confidence Stress Management Memory & Study Skills Motivation Other Personal Development
Design
Web Design Graphic Design & Illustration Design Tools User Experience Design Game Design 3D & Animation Fashion Design Architectural Design Interior Design Other Design
Marketing
Digital Marketing Search Engine Optimization Social Media Marketing Branding Marketing Fundamentals Marketing Analytics & Automation Public Relations Paid Advertising Video & Mobile Marketing Content Marketing Growth Hacking Affiliate Marketing Product Marketing Other Marketing
Lifestyle
Arts & Crafts Beauty & Makeup Esoteric Practices Food & Beverage Gaming Home Improvement & Gardening Pet Care & Training Travel Other Lifestyle
Photography & Video
Digital Photography Photography Portrait Photography Photography Tools Commercial Photography Video Design Other Photography & Video
Health & Fitness
Fitness General Health Sports Nutrition & Diet Yoga Mental Health Martial Arts & Self Defense Safety & First Aid Dance Meditation Other Health & Fitness
Music
Instruments Music Production Music Fundamentals Vocal Music Techniques Music Software Other Music
Teaching & Academics
Engineering Humanities Math Science Online Education Social Science Language Learning Teacher Training Test Prep Other Teaching & Academics
Web Development JavaScript React Angular CSS Node.Js PHP HTML5 Vue JS
AWS Certification Microsoft Certification AWS Certified Solutions Architect - Associate AWS Certified Cloud Practitioner CompTIA A+ Amazon AWS Cisco CCNA Microsoft AZ-900 CompTIA Security+
Microsoft Power BI SQL Tableau Data Modeling Business Analysis Business Intelligence MySQL Qlik Sense Data Analysis
Unity Unreal Engine Game Development Fundamentals C# 3D Game Development C++ Unreal Engine Blueprints 2D Game Development Blender
Google Flutter iOS Development Android Development Swift React Native Dart (programming language) Kotlin Mobile App Development SwiftUI
Graphic Design Photoshop Adobe Illustrator Drawing Digital Painting Canva InDesign Character Design Procreate Digital Illustration App
Life Coach Training Neuro-Linguistic Programming Personal Development Personal Transformation Life Purpose Mindfulness Sound Therapy Meditation Emotional Intelligence
Business Fundamentals Entrepreneurship Fundamentals Freelancing Business Strategy Startup Business Plan Online Business Blogging Leadership
Digital Marketing Social Media Marketing Marketing Strategy Google Analytics Internet Marketing Email Marketing Copywriting YouTube Marketing Startup
2022-05-04T18:06:16Z

IT & SoftwareOther IT & SoftwareAzure Data Factory

Azure Data Factory for Beginners - Build Data Ingestion

Learn Azure Data Factory by building a Metadata-driven Ingestion Framework as an industry standard
Rating: 4.4 out of 54.4 (313 ratings)
3,221 students
Created by David Charles Academy
Last updated 11/2021
English
English [Auto]

What you'll learn

  • Azure Data Factory
  • Azure Blob Storage
  • Azure Gen 2 Data Lake Storage
  • Azure Data Factory Pipelines
  • Data Engineering Concepts
  • Data Lake Concepts
  • Metadata Driven Frameworks Concepts
  • Industry Example on How to build Ingestion Frameworks
  • Dynamic Azure Data Factory Pipelines
  • Email Notifications with Logic Apps
  • Tracking of Pipelines and Batch Runs
  • Version Management with Azure DevOps

Requirements

  • Basic PC / Laptop

Description

The main objective of this course is to help you to learn Data Engineering techniques of building Metadata-Driven frameworks with Azure Data Engineering tools such as Data Factory, Azure SQL, and others.


Building Frameworks are now an industry norm and it has become an important skill to know how to visualize, design, plan and implement data frameworks.


The framework that we are going to build together is referred to as the Metadata-Driven Ingestion Framework.


Data ingestion into the data lake from the disparate source systems is a key requirement for a company that aspires to be data-driven, and finding a common way to ingest the data is a desirable and necessary requirement.


Metadata-Driven Frameworks allows a company to develop the system just once and it can be adopted and reused by various business clusters without the need for additional development, thus saving the business time and costs. Think of it as a plug-and-play system.


The first objective of the course is to onboard you onto the Azure Data Factory platform to help you assemble your first Azure Data Factory Pipeline. Once you get a good grip of the Azure Data Factory development pattern, then it becomes easier to adopt the same pattern to onboard other sources and data sinks.


Once you are comfortable with building a basic azure data factory pipeline,  as a second objective we then move on to building a fully-fledged and working metadata-driven framework to make the ingestion more dynamic, and furthermore, we will build the framework in such a way that you can audit every batch orchestration and individual pipeline runs for business intelligence and operational monitoring.


Creating your first Pipeline


What will be covered  is as follows;

1. Introduction to Azure Data Factory

2. Unpack the requirements and technical architecture

3. Create an Azure Data Factory Resource

4. Create an Azure Blob Storage account

5. Create an Azure Data Lake Gen 2 Storage account

6. Learn how to use the Storage Explorer

7. Create Your First Azure Pipeline.


Metadata Driven Ingestion


1. Unpack the theory on Metadata Driven Ingestion

2. Describing the High-Level Plan for building the User

3. Creation of a dedicated Active Directory User and assigning appropriate permissions

4. Using Azure Data Studio

5. Creation of the Metadata Driven Database (Tables and T-SQL Stored Procedure)

6. Applying business naming conventions

7. Creating an email notifications strategy

8. Creation of Reusable utility pipelines

9. Develop a mechanism to log data for every data ingestion pipeline run and also the batch itself

10. Creation of a dynamic data ingestion pipeline

11. Apply the orchestration pipeline

12. Explanation of T-SQL Stored Procedures for the Ingestion Engine

13. Creating an Azure DevOps Repository for the Data Factory Pipelines


Event-Driven Ingestion

1. Enabling the Event Grid Provider

2. Use the Getmetadata Activity

3. Use the Filter Activity

4. Create Event-Based Triggers

5. Create and Merge new DevOps Branches



Who this course is for:

  • Aspiring Data Engineers
  • Developers that are curious about Azure Data Factory as an ETL alternative

Instructor

David Charles Academy
Senior Big Data Engineer / Consultant at ABN AMRO
David Charles Academy
  • 4.4 Instructor Rating
  • 894 Reviews
  • 18,442 Students
  • 7 Courses

I'm a Data Management professional that is driven by the power and influence of data in our lives. With the power of data, I was able to help companies become more data-driven to gain a competitive edge or meet regulatory requirements.


In the last 15 years, I have had the pleasure of designing and implementing data warehousing solutions in Retail, Telco, and Banking industries, and recently in more big data lake specific implementations.


I’ve had the pleasure to be lead and also lead teams to implement the above-mentioned strategies and in my spare time, I teach programming online as a YouTuber as I am passionate about technology.

Top companies choose Udemy Business to build in-demand career skills.
NasdaqVolkswagenBoxNetAppEventbrite
  • Udemy Business
  • Teach on Udemy
  • Get the app
  • About us
  • Contact us
  • Careers
  • Blog
  • Help and Support
  • Affiliate
  • Investors
  • Impressum Kontakt
  • Terms
  • Privacy policy
  • Cookie settings
  • Sitemap
  • Accessibility statement
Udemy
© 2022 Udemy, Inc.