Azure Data Factory for Beginners - Build Data Ingestion

Explore metadata driven ingestion in Azure Data Factory to process multiple files with one pipeline, reducing maintenance. A metadata database guides the orchestration and dynamic ingestion into the data lake.

Submit a metadata-driven one-off batch ingestion, initialize the batch, and run ingestion from Azure Blob storage into the Data Engineering Data Lake using an Azure Data Factory pipeline.

Create an Azure Active Directory user to manage Azure resources securely, avoiding the high-privilege super user. Set an initial password and usage location (region) before proceeding to assign permissions.

Assign the contributor role to a user via access control on an azure subscription, granting permissions to create resources and manage access.

Disable security defaults in Azure Active Directory to skip mandatory multi-factor authentication for the course exercise, by navigating to Azure Active Directory, properties, and manage security defaults, then save.

Create an Azure SQL database to store metadata for batch pipeline management, configure a basic 2 gigabyte database in the data engineering resource group, and enable access via firewall rules.

Install Azure Data Studio to manage your Azure SQL databases by downloading the appropriate installer and connecting with the server name and SQL login.

Create metadata tables and stored procedures in a metadata database using SQL scripts, Azure Data Studio, and initial data population to enable data ingestion.

Reconfigure the existing data factory by applying new business naming conventions, creating new linked services for Azure Blob Storage, renaming datasets, testing connections, and publishing changes.

Set up a logic app to send email notifications from an Azure Data Factory pipeline by handling HTTP requests, generating a JSON schema, and wiring Gmail as the email server.

Modify Azure data factory pipeline to trigger email notifications via a logic app using a web activity with http post, sending a json payload containing subject and body, then test.

Create a linked service to the metadata database using Azure SQL and test the connection. Create a SQL dataset for the email recipients table, name ds_sql_email_addresses, preview results, and publish.

Create a reusable utility pipeline in Azure Data Factory to send email notifications by calling get email addresses and looping with for each to deliver messages.

Explore how the email recipients table links to the system info table using a system id foreign key, enabling targeted notifications by system and dataset.

Explore how the get email addresses stored procedure in azure data studio retrieves email recipients using system code and dataset name, showing a two-parameter SQL procedure and its where clause.

Clone the ingestion pipeline, add system code and dataset name parameters, and replace the web activity with an execute pipeline to the email utility.

Track and log every triggered pipeline in Azure Data Factory by cloning the pipeline, adding run id and snapshot date, calling insert pipeline log, and sending email notifications.

Enable dynamic subject and message parameters in the email notification pipeline, and use a switch to send status-based notifications (success, failed, default).

Make logging dynamic with a standalone utility pipeline to record both success and failure metadata in the pipeline log table, using parameters like snapshot date, status, and run.

Learn to extend the azure data factory ingestion pipeline by logging pipeline metadata on success or failure, using variables and time zone aware timestamp formatting.

Modify the log pipeline metadata to send messages only when an ingestion pipeline fails, using an if condition to suppress success notifications.

Make datasets metadata driven by replacing hardcoded json sources with runtime parameters for container, directory, and file name, and use cloning to read from folders with dynamic content.

Observe source-to-target details and upload missing files to azure blob storage. Clone and version the ingestion pipeline, then configure a metadata-driven lookup with a stored procedure for the copy activity.

Learn to implement an iterative ingestion workflow in Azure Data Factory using for each activity, lookup, and copy data steps to move from source to target and log metadata.

Explain how the source to target info stored procedure feeds the ingestion pipeline using system code and stage, mapping data to raw, trusted, and curated lake layers.

Create an orchestration pipeline to drive daily batch runs by retrieving batch details from the metadata database via a lookup using the get_batch stored procedure.

Attach a for-each loop to the lookup results, run ingestion pipeline version six for each batch, and set batch status after completion.

Fix duplicated batch ingestions by adding a dataset filter to the source-to-target info procedure, cleaning the pipeline log, and validating the orchestration pipeline in Azure Data Factory.

Explore the pipeline log and tables to track run IDs and batches, measure latency and duration, and audit data ingestion into the data lake via the source to target view.

Understand how the get batch stored procedure initializes ingestion batches. It handles time zones, timestamps, and status transitions for not started, in progress, and completed with errors, including restart logic.

Explore how the set batch status stored procedure updates batch status after ingestion and generates run id with a snapshot date, and how get run id stitches a run id.

Set up an Azure DevOps Git repository to store code and metadata SQL scripts for your metadata-driven pipelines, with a readme, folders, and commits to the main branch.

Clean up unused data factory pipelines, validate changes, and publish the data factory to Azure DevOps Git by configuring branches for main and collaboration.

Finish version one of the generic ingestion platform and plan to extend it for databases or files such as json, csv, parquet, and xml with a data factory approach.

Explore event-driven ingestion with the Azure Data Factory as finance container uploads trigger automatic pipelines. Ingest only csv files, finance.csv and department.csv, into separate folders in the data lake.

Configure reading from azure blob storage by creating the finance container, uploading finance csv and accounting json files, and building a data factory dataset to list container contents as metadata.

Create a finance container in the web store storage account, download and extract the event driven zip, and upload the three files to prepare a dataset in Data factory.

Create a new dataset in Azure Data Factory Studio to read CSV files from the finance container in Azure Blob Storage, using a delimited text format and a container parameter.

Set up real-time Azure Data Factory pipeline to write finance and department data into a timestamped data lake, and define a sink dataset with container, directory, and file name.

Create a sink dataset in azure data factory by configuring a azure data lake storage gen2 csv dataset, linking to rls_data_engineering, with finance container, directory, and finance.csv.

Plan a data factory pipeline that reads from Azure Blob storage using source container and file type parameters, filters by metadata, ingests CSV files into the data lake.

Learn to filter Azure Data Factory pipelines to CSV files using a Get Metadata output, an ends-with check, and a Filter activity, preparing CSV ingestion to the data lake.

Clone the csv metadata dataset, add a file name parameter, and create a dataset that reads each csv file to ingest into the data lake.

Ingest csv files from the pilot finance container into the data lake with a for each file loop and copy data actions driven by the file type filter.

Enable the event grid feature and configure an event-based trigger in Azure Data Factory. Test ingestion by deleting and uploading a csv in the finance container to trigger the pipeline.

Enable the event grid provider to support event-based triggers for data factory pipelines. Register the provider to monitor Azure artifacts for events and enable responsive workflows.

Delete csv files in the finance and department containers and clear the raw data in the data lake to prepare an event-driven ingestion test; set up the event-based trigger.

Create an event based trigger in Azure Data Factory by selecting storage events, monitoring the finance container for blob created events, and attaching it to a pipeline.

Publish code to the main branch by creating a pull request, merge changes, publish the pipeline, and start the trigger, then monitor the event grid for at least ten minutes.

Verify event driven ingestion by triggering pipelines in Azure Data Factory as you upload files to the storage account, and confirm the raw finance data lands in the data lake.

Closing remarks recap building an event-driven data ingestion pipeline with metadata and filter steps, and outline extending it to add file audit, pipeline logging, alerts, and multi-file support beyond CSV.

Learn how infrastructure as code enables reproducible, automated provisioning of Azure data platforms using ARM, Bicep, and pipelines, with Azure DevOps, Git, and Data Factory.

Register a free Microsoft Azure account to access 12 months of use and about 200 USD credit for the first 80 days, with identity verification and credit card sign-up.

Create an Azure DevOps organization, set up a project named daybreak with a git repository to host code and plan work, and create a pdb-provision-resource-groups repository for infrastructure as code.

Install the azure cli across platforms, learn to run commands in windows powershell, cmd, or git bash, and verify installation by checking the version and signing into your azure tenant.

install and verify the bicep language in Windows PowerShell to enable declarative infrastructure as code for Azure resources, using az bicep install and az bicep --version (0.12.40).

Install Git Bash on Windows to access a Bash terminal, use Git for versioning, and work with Unix-like syntax for project artifacts.

Install Visual Studio Code as code editor for infrastructure as code and Git Bash. Download Windows installer from Google, run it, accept license, and prepare for ARM and Bicep extensions.

Install essential Visual Studio Code extensions for Azure Data Factory beginners, including Bicep, Azure Resource Manager, PowerShell, and YAML, to enable validation, IntelliSense, and fewer errors.

Set git bash as the default terminal in Visual Studio Code using the command palette to select default profile, then open a new terminal to verify.

Git is a distributed open source version control system that stores code, tracks history, and supports cloning from a master repository, branching, committing, pushing, and merging for collaboration.

Create a local git repository by making a git demo folder, navigating into it, and running git init to version-control your code.

Create a demo file with initial content, stage it with git add, commit to the master branch, and configure global user name and email.

Stage and commit multiple files in git using the period with git add, then verify with git status and commit with a concise message.

Learn how to create and switch to a new branch from the master to isolate code changes, test, and later merge back into the main branch.

Modify an existing file in the new branch, stage changes, and commit with a message; verify history with git log and status, preparing to merge in the next lecture.

Merge the pr test branch into master to demonstrate branch isolation, create and commit a new file, and verify the merge using git commands and Visual Studio Code.

Learn to manage Git using Visual Studio Code, including creating a new branch, staging changes, and committing to master. Merge branches and explore basic Git workflows in Visual Studio Code.

Learn to view a project's past state with git by using one-line logs, checking out a specific commit, inspecting files, and returning to master to undo changes.

Learn to undo changes in git by inspecting the head revision, staging edits, and applying git revert, with practical Visual Studio Code steps and the demo nine.txt updates.

Explore time traveling into the past with git by using log and checkout, then apply soft and hard resets to manage commits and unstaged files.

Set up billing for your Azure DevOps organization and configure a Microsoft hosted runtime agent to run and test pipelines, including free minutes and self-hosted options.

Install the net framework and core SDKs to prepare a self hosted runtime agent for Azure DevOps pipelines, enabling application deployment and runtime services.

Install PowerShell 7 to avoid pipeline script compatibility issues and use built-in Azure libraries shared across PowerShell versions. Follow downloading and installing the 64-bit Windows installer.

Install Visual Studio 2022 Community Edition to access Azure development libraries and ASP.NET web development tools for creating and deploying web applications and Azure resources, and configure a self-hosted agent.

Set up a personal access token in Azure DevOps, set a 90-day expiration with full access, then securely save the token and prepare the self-hosted agent to run pipelines.

Download a 64-bit Windows self-hosted runtime agent and configure it in the default Azure DevOps pool using PowerShell and the config.cmd, with a personal access token.

Test a self hosted agent in Azure DevOps by running a pipeline, selecting the default pool, and reviewing live logs to confirm the agent executes as expected.

Create a service connection in Azure DevOps to authenticate against Microsoft Azure, automatically provisioning a service principal linked to your Azure subscription and configuring resource manager settings.

Clone your git repository into a local workspace, authenticate with your credentials, and open it in Visual Studio Code to prepare the project structure for the next lecture.

Organize your Azure Data Factory project by establishing a clear structure with pipelines and scripts folders, creating a provision-resource-group yaml pipeline and a create-resource-groups.ps1 PowerShell script.

Write a PowerShell script to provision an Azure resource group with tagging, accepting location, resource group name, owner, cost center, application, and description in Visual Studio Code.

Create a yaml-driven pipeline by adding a stage and a job with steps to provision resource group yaml, set trigger none, then commit and push to origin in Azure DevOps.

Add a final task to an Azure DevOps YAML pipeline to run a PowerShell script that provisions an Azure resource group, configuring the Azure PowerShell task and script path.

Validate the provisioned resource group in the portal, then merge provisioning code into main via a pull request in Azure DevOps, and pull latest changes before moving to Bicep.

Create an Azure DevOps repository and project structure to provision infrastructure and enable log analytics, organizing directories and files for infrastructure code and artifacts.

Set up a repository for provisioning infrastructure in DevOps, create a new repository named p d B infrastructure, then prepare to clone it locally and create a working git branch.

Clone the repository to your local computer, create a working Git branch for provisioning log analytics, and launch Visual Studio Code to prepare the project structure.

Create a project structure with infra, log analytics, and modules folders and a main.bicep file. Add a pipelines directory with a YAML file and a scripts folder for Bicep compilation.

Learn to build a reusable log analytics bicep module as infrastructure as code and modify the main bicep file to deploy log analytics across multiple resource groups using modular programming.

Define a log analytics bicep module template by specifying required and optional inputs, including name, location, tags, sku, log retention, and public network access for ingestion and query.

Declare and annotate input parameters in the Log Analytics Bicep template, including tags, location, workspace name, retention in days, skew, and network access for ingestion and query.

Develop a log analytics bicep module template by defining input properties, using the 2021-1201 preview API, and wiring outputs for the workspace name and id.

Develop the main bicep file for Log Analytics by defining input sections for tagging, resource naming, and Log Analytics parameters, then invoke the Log Analytics module template to drive provisioning.

Create tagging parameters and a naming convention in bicep, building a tag object with owner, cost center, application, description, and repository, and define log analytics naming using application and environment.

Add parameter variables for the log analytics module, establish a naming convention; copy from the template, set location from the resource group, and build the workspace name with parameter references.

Invoke the log analytics bicep module template using a module declaration command with a unique deployment name and parameters including log analytics workspace name, retention days, and public network access.

Develop a reusable PowerShell and CLI utility to transpile Bicep to ARM JSON, enabling Azure Resource Manager to deploy declarative configurations.

Plan a script that transpiles Bicep to an ARM template, using a file path, template path, and output file name, producing a JSON template while maintaining the latest Bicep library.

Build a PowerShell script to compile Bicep to ARM JSON, with parameters for file path, Bicep template, and output file, plus try-catch error handling and debugging output.