Azure Data Factory For Data Engineers - Project on Covid19

Explore Azure Data Factory as a fully managed serverless data integration solution that ingests, prepares, and transforms data at scale with 90+ connectors across multi-cloud, on-prem, and SaaS sources.

Build data platform for EU countries and the UK with a data lake and data warehouse to enable data scientists to run machine learning models predicting Covid-19 spread and trends.

Ingest covid-19 and eurostat population data. Using azure data factory, connect via http; transform with data flow, HDInsight, and Azure Databricks; store in data lake and SQL for Power BI.

Explore Azure storage solutions overview and learn how data structure and needs guide selection among blob, file, disk, table, and queue storage, plus Data Lake Storage Gen2 and Cosmos DB.

Set up the course environment by provisioning seven resources: Azure subscription, Data Factory with data flow, Blob Storage, data lake, SQL Database, and Databricks and HD Insight clusters.

Create a free Azure account and access 12 months of free services plus 150 pounds or 200 USD credits within 30 days, with guidance on Microsoft sign-in and student differences.

Get an overview of the Azure portal for accessing resources, sign in at portal.azure.com, and navigate with search, cloud shell, subscriptions, and notifications.

Create an Azure data factory in the portal, configure subscription, region, and resource group, then explore author and monitor, IAM, and dashboard pinning for the covid reporting project.

Create an Azure storage account in the portal for the covid-reporting-rg in UK South, using the standard tier and LRS replication, then manage access with IAM and use access keys.

Learn to use the Azure Storage Explorer desktop to manage storage accounts, upload files and folders, switch between subscriptions, and interact with containers like Windows Explorer.

Create an Azure Data Lake Storage Gen2 account in the portal, enabling hierarchical namespace to unlock big data analytics on top of standard storage.

Explore ingesting EU population data from Azure Blob Storage to a data lake with Azure Data Factory, covering copy activity, link services, datasets, pipelines, control flow, triggers, and best practices.

Ingest data from Azure blob storage to Azure Data Lake using copy activity in Azure Data Factory, outlining requirements and building blocks like link services, datasets, and pipelines.

Prepare the environment by creating a blob container named population, uploading the file, and creating a raw container in the data lake; then proceed to configure data factory components.

Establish naming standards for an Azure Data Factory pipeline, detailing abbreviations for link services, data sets, and pipelines, and naming conventions for ADLS, blob storage, and gzipped raw covid data.

Create link services for Azure Blob Storage and Data Lake Storage Gen2, then create datasets, assemble a pipeline with a Copy activity, and debug end-to-end in Azure Data Factory.

Create an Azure Data Factory pipeline to copy data from blob storage to a data lake using a copy data activity, with link services, datasets, source and sink, and validation.

Learn to use the validation control flow activity in Azure Data Factory to wait for a file arrival, with a 24-hour timeout and 10-minute checks, creating a production-ready pipeline.

Use get metadata and an if condition to verify the input file has 13 columns before copying to the data lake, and trigger a web activity to alert on mismatch.

Link a delete activity to the copy activity in Azure Data Factory to remove the source file after a successful copy, since move isn't available.

Learn how Azure Data Factory uses schedule, tumbling window, and event based triggers to run pipelines at set times or in response to file events, with UTC time handling.

Choose an event-based trigger for daily blob arrivals, delete after processing, and register the event grid resource, while noting schedule triggers as an alternative.

Explore the recent changes to ECDC data, including moving from daily to weekly updates and week-numbered files, and follow the git repository and URLs in your data factory workflow.

Ingest COVID-19 data from the ECDC website into a lake with Azure Data Factory, building a metadata driven pipeline with variables, parameters, and lookup and foreach to process four files.

Ingest COVID-19 data from the ECDC into Azure Data Lake using Azure Data Factory, daily downloads of cases, deaths, hospital and ICU admissions, testing, country response measures, with data dictionaries.

Design a daily data factory pipeline that ingests a CSV of cases and deaths from an HTTP URL into Azure Data Lake Gen2 using a copy activity.

Learn to create dynamic pipelines in Azure Data Factory by parameterizing the relative URL and file name, using parameters and variables to reuse a single pipeline for multiple files.

Parameterize the pipeline to replace variables with runtime inputs for source URL and sink file name, enabling a generic ECDC data ingest via a schedule trigger.

Learn to build an Azure Data Factory pipeline using a lookup to read a json config and a for-each loop to copy multiple datasets with a single trigger.

Parametrize the http link service by adding a source base URL, propagate it through dataset and pipeline parameters, and validate the changes with a debug run before publishing.

Process four files with a single metadata driven pipeline using a JSON config, lookup, foreach, and copy with a schedule trigger in Azure Data Factory.

Learn to transform COVID-19 cases and deaths data ingested from the ecdc site into a data lake using azure data factory data flows, building a pipeline with key transformations.

Explore mapping data flows in Azure Data Factory to visually design transformations, with Spark execution on Databricks clusters and notes on region and connector limitations.

Explore the Azure Data Factory data flow UI, create a mapping data flow, configure source and sink transformations, and understand partitioning, data preview, and debugging with Spark clusters.

Transform Europe-only cases and deaths data from the data lake by normalizing country codes with a lookup, deriving confirmed cases and deaths counts, and renaming dates to reported date.

Enable debug and spark integration, build a mapping data flow with a source transformation using a data lake gen2 CSV dataset, and test with sampling and schema drift.

Apply a filter transformation in Azure Data Factory to isolate Europe data by continent equals Europe and non-null country codes.

Use the select transformation to drop the continent and rate 14 day fields, rename date to reported date, and explore fixed versus rule-based mappings with auto-mapping and skip-duplicate options.

Pivot the daily count by indicator using the pivot transformation in Azure Data Factory, summing counts to produce confirmedcases_count and deaths_count by grouping country, country code, population, and reported date.

Learn how to use a lookup transformation in Azure Data Factory to enrich covid-19 case and death data with a 2-digit country code from a CSV lookup.

Configure a sink transformation in Azure Data Factory to write covid-19 cases and deaths to a csv file in Data Lake Storage Gen2, using a dataset.

Create Azure Data Factory pipeline to run a data flow, configure a spark cluster, debug and trigger executions, monitor results, and publish a single-file cases-and-deaths output to the processed folder.

Transform hospital admissions data from ecdc in the data lake using data flows, with transformations like source, select, look at, pivot, and sync, then run it via a pipeline.

Transform the raw hospital admissions data by splitting daily and weekly files, creating dedicated occupancy count columns, and deriving week start and end dates for easy analysis.

Create a data flow with a source transformation to read the hospital admissions CSV from raw data lake storage, configure ds_raw_hospital_admissions dataset with a first row header and fixed schema.

Remove the URL field and rename date to reported date, and week to reported year and reported week within a select transformation in the data flow.

Perform a country lookup to fetch 2-digit and 3-digit country codes and population, then apply a select transformation to drop continent and duplicate inputs and outputs in the data flow.

Learn to route hospital data into daily and weekly streams using Azure Data Factory's conditional split, defining weekly and daily conditions to produce two outputs.

Join the weekly hospital admissions data with the dim date lookup in Azure Data Factory to derive week start and end dates.

Create an ECDC year, week column in the data flow via the derived column transformation to derive year and a two-digit week in the format 'YYYY- W 01' using LPAD.

Create weekly records from the date file using an aggregate transformation with a group by on ECDC year week, computing week start and end dates and a derived week number.

Join the hospital admissions stream with the aggregate dim date source, using join transformation to add week start and end dates with an inner join on year week key.

Perform pivot transformations on daily and weekly streams, group by relevant columns, and sum values to derive hospital and ICU occupancy counts from the indicator and value fields.

Apply the sort transformation to hospital admissions data, sorting by reported year and week descending, then by country ascending, with options for case sensitivity and partitioning.

Select the fields for the weekly and daily files and write all to csv sinks in Data Factory's Data Lake Gen2, enabling the subsequent pipeline workflow.

Create and publish an ADF pipeline to run the hospital admissions dataflow, then trigger and monitor it to validate processing. Ensure daily and weekly partitions are written to single files.

Prepare data for Azure Data Factory with HDInsight and Spark by writing outputs to folders, and update configs for ECDC data, cases and deaths, and dim date and dim country.

Transform the testing file from the ECDC into your data lake using the HDInsight activity, with a Hive script, for analysis and reporting, and build a pipeline to verify outputs.

Create an HDInsight cluster in the Azure portal by provisioning a user-assigned managed identity and granting blob data owner access to the Data Lake Gen2.

Explore the hdinsight ui and ambari to manage clusters, monitor with ambari, and run hive queries. Learn to configure access control, scaling, quotas, and jdbc connections for data factory workloads.

Transform the ECDC testing data in the data lake by enriching it with two- and three-digit country codes, week start and weekend dates, cases, tests, and positivity rate.

Explore a Hive script to transform covid data in HDInsight. Create lookup, raw, and process data databases, external and managed tables, and an insert-based pipeline for data factory processing.

Create an Azure Data Factory pipeline that uses a Hive activity to run a script on an HD Insight cluster, linking storage and Ambari to produce and view results.

Explore HDInsight activities in Azure Data Factory, including Pig, MapReduce, Spark, and streaming, and learn how to delete the cluster to avoid charges after finishing the project.

Learn to use the Databricks notebook activity in Azure Data Factory to transform population data from Azure Block Storage into a data lake, and run a pipeline to verify outputs.

Create an Azure Databricks service in the Azure portal, then create a Databricks cluster and mount the data lake storage to enable transformations for Data Factory.

Create an Azure Databricks interactive cluster for covid reporting with a single node DS3_v2, 7.3 LTS runtime, and 20-minute idle termination, enabling notebook-based ETL and Azure Data Factory integration.

Mount the data lake in Databricks by creating an Azure service principal and mounting raw, processed, and lookup containers with the mount_storage.py script.

Transform population by age data from the data lake by filtering to 2019, splitting country codes, pivoting age groups into columns, and joining with the country dim.

Create an Azure Data Factory pipeline that executes a Databricks notebook activity via a link service, transforming population data with a job cluster and access token.