ETL on Cloud Platforms Azure, AWS & GCP

Discover how cloud computing delivers on-demand resources via the internet with pay-as-you-go pricing, scalable services, global reach, and the key IaaS, PaaS, SaaS models.

Explore the ETL process, extract, transform, and load, across Azure, AWS, and GCP, with emphasis on orchestration, scheduling, monitoring, data lineage, and loading into data warehouses for analysis.

Navigate the Azure portal and dashboard, manage resource groups, explore the marketplace’s analytics offerings like data factory and data bricks, and use Azure Monitor and Cloud Shell.

Navigate the Google Cloud Platform portal to manage projects, billing, IAM, storage, BigQuery, Data Fusion, and Cloud Shell within the dashboard and navigation.

Explore the AWS management console and its key features, including dashboards, service navigation, IAM, CloudWatch, CloudTrail, CloudFormation, and S3 workflows for configuring, monitoring, and securing resources.

Set up a resource group and blob storage in Azure, upload Zambian national parks data, and inspect file contents and metadata to prepare for next-step etl development.

Build two azure etl pipelines using copy activities to load, consolidate, and enrich files with metadata, then load a single file into sql; organize in resource group and blob storage.

Upload data to an Azure Blob Storage container, create folders and organize files, then create an Azure Data Factory instance to build ETL pipelines.

Design an Azure data factory etl pipeline with two pipelines, using a metadata activity, datasets, and a copy activity to move csv files from input to output blob storage.

Azure copy activity ETL pipeline part 2 shows cleaning a malformed csv file, removing bad inputs, correcting extensions, and merging outputs into a single csv for downstream use.

Create and configure an Azure SQL database and server, set the admin, open the firewall, and connect with SQL Server Management Studio to support development ETL workflows.

Create a sql database user for Azure Data Factory and Databricks, define a schema and table, and verify connections to enable the end-to-end etl from blob storage to the database.

Connect and configure an Azure Data Factory pipeline to copy CSV data from blob storage to a SQL database. Map fields and apply a pre copy script to prevent duplicates.

Learn how to design an AWS ETL using S3 inputs and a destination bucket, powered by Glue, with IAM controls, crawlers, and the Glue catalog to manage databases and tables.

Create an S3 bucket with input and output folders, then upload the data, and configure an IAM role to enable access for Glue and S3 in the next lesson.

learn to create an AWS IAM role for Glue, distinguish role from user, assign a service policy (including S3 access), and plan cleanup via deletion.

Build and manage an end-to-end AWS Glue ETL workflow by creating input and output databases sourced from S3, configuring crawlers, and exploring the data catalog and ETL engine.

Create an AWS Glue ETL job using data catalog as a pointer and IAM role to access data in S3; split park into pack size and units, parquet outputs.

View and verify the etl data load by running the output crawler, creating the output table, and confirming park name and geolocation via the glue catalog and s3 linkage.

Develop an ETL on Google Cloud Platform using Cloud Storage, Data Fusion, and BigQuery to extract and load data, while creating an umbrella project named Zambian National Parks.

Create and configure a Google Cloud Storage bucket, then upload the ETL data, following a descriptive naming convention and privacy settings, before exploring data fusion for ETL.

Explore Google Cloud Data Fusion, a fully managed no-code ETL service, and build a batch pipeline to load CSV data from Cloud Storage into BigQuery using Wrangler.

Wrap up this course on cloud platforms, including Azure, AWS, and GCP, by reflecting on ETL in the cloud and the data integration skills gained.