AWS Glue - The Complete Masterclass

Explore identity and access management concepts, create user groups and policies, and implement encryption with CMS keys, plus set up server-side and client-side encryption and an SNS topic for notifications.

Explore identity and access management in AWS, detailing authentication, authorization, and identities (users, groups, roles). Learn how policies define allowed actions on resources like glue jobs.

Create IAM users in the AWS console, enable programmatic access with an auto-generated password, then form a user group with an administrative policy and assign the user for CLI access.

Explore the key management service by enabling encryption at rest and in transit, compare client-side and server-side encryption, and create a symmetric customer-managed key with a resource policy.

Create and configure an AWS SNS topic and email subscription to notify on AWS Glue job failures, using standard messaging and subscription confirmation.

Recap of section z covers creating an IAM user and group with administrator access, policies and keys, an S3 encryption key, an SNS topic, and a Glue job role.

Learn sd 101 concepts, anchor data in a demo bucket with a cms key, adjust bucket policies, and explore cli 101 and cloud formation 101 basics.

Learn aws s3 101 by creating a demo bucket with a unique name, enabling encryption with cms, blocking public access, and applying deny policy that allows only a specific role.

Configure the AWS CLI using IAM user credentials, generate and manage access keys, and verify the connection before exploring commands for IAM, cloud formation, and S3 services.

Explore how cloud formation uses templates as infrastructure as code to create, update, and delete AWS resources with JSON templates, enabling consistent, parameterized deployments across environments via stacks.

Download, unzip, and review the source data files containing ETL artifacts, scripts, cloud forms, and template sources, then create and populate the ETL, artifact, and cloud formation buckets for deployment.

Create and configure three S3 buckets for AWS Glue workflows: a template bucket for Glue source data, an ETL artifacts bucket, and a crawler data bucket for lab data sources.

Upload data to s3 buckets by selecting the cloud data bucket, uploading a folder of cloud files, and managing artifacts in the artifacts bucket for the glue job templates.

Create your first glue database and explore the glue data catalog through hands-on labs. Schedule and orchestrate glue jobs with triggers and workflows to visualize the end-to-end pipeline.

Explore the aws glue data catalog essentials, including databases, tables, partitions, and connections, and learn how metadata supports etl jobs and redshift workflows.

Learn how AWS Glue crawler accesses data sources, extracts metadata, creates and updates table definitions in the cloud catalog, and manages schemas and partitions with classifiers as needed.

Explore how AWS Glue crawler classifiers infer schemas and create tables in the Glue catalog, and build custom classifiers to handle csv, json, xml with grok patterns.

Run the first AWS Glue crawler, monitor the cloud watch logs, and view the created CSV and JSON tables in the data catalog, confirming the schema and metadata.

Drive a second AWS Glue crawler lab to create and update a table in the AWS Glue catalog, expanding from three to six columns as json data uploads occur.

Create and run a third AWS Glue crawler for an S3 CSV data source, creating a prefixed table in database 101.

Create an AWS Glue crawler to ingest historical data while excluding current data, generate historical and historical year tables with partitions by extracted date and year, enabling partition-aware queries.

Create and run an AWS Glue crawler to catalog current data from an S3 bucket while excluding historical folders, resulting in a new current table without partitions.

Create and configure AWS Glue jobs for ETL with script location, artifact buckets, cloud formation, and optional libraries; choose Python shell or Spark jobs, set bookmarks, parameters, and autoscale.

Discover how AWS Glue workflows orchestrate triggers and jobs to manage complex ETLs, monitor each step, and diagnose failures, with the ability to stop and resume as needed.

Review CloudFormation template 101, update and upload the template, adjust account-specific paths, and use the AWS CLI to push updated templates to the bucket while debugging stack failures.

Explore CloudFormation templates 101 to create and update AWS Glue resources, including jobs, roles, and workflows, update the S3 bucket pad, and use JSON and YAML templates using the console.

Learn to build the first glue pipeline with CloudFormation templates, provisioning a glue job resource (spark or python) and an IAM role, with script location and bucket permissions.

Build a second glue pipeline with CFN templates to create a crawler, glue job, and a workflow with scheduled and conditional triggers for automated data processing.

Upload cloud formation templates to S3 using the AWS CLI, syncing local templates to the target bucket and organizing them in a templates folder for future Glue jobs.

Deploy your first AWS Glue blue job by preparing source and target, using CloudFormation templates, debugging deployments, and validating results via job output and CloudWatch logs.

Examine how to run a glue job script using glue and spark contexts, convert between dynamic frames and data frames, apply transformations, and write output to the target bucket.

Inspect the job output by reviewing output logs, verifying printed data frames, and using logger.info to confirm logs, while understanding glue log groups and cloud watch access.

Align for deployment, deploy the AWS Glue job, fix the script despite failures, and ensure the AWS Glue job workflow runs the job and crawler to create data catalog table.

Debug a glue job by reading script parameters, processing a header CSV with country, city, and year filters, and writing partitioned results to destination bucket; fix launch and access errors.

Prepare the required S3 buckets and artifacts for the three glue streaming pipelines, ensure the transformation input and lookup files exist, and verify the cloud formation template to provision resources.

Deploy a cloud formation stack to provision AWS Glue job resources, roles, an AWS Glue catalog database, and a JSON streaming table with a Kinesis stream for real-time data.

Create a Python shell Glue job to generate Kinesis real-time streaming records from a CSV in S3, then publish to a Kinesis stream for downstream Glue jobs.

Design a Glue streaming loader job to read from a Kinesis stream and write to an S3 bucket, using a visual canvas and a shared script for subsequent jobs.

Recap of three aws glue streaming jobs: a python cell job reads csv from kinesis, a loader writes to s3, and a transformer builds a final frame with country codes.

Launch a spark streaming Glue transformation to read streaming data, update country names from codes, and write CSV output to the transformation output location with a kinesis checkpoint.

Discover three AWS Glue streaming jobs: a Kinesis-based CSV reader, a streaming loader loading data to S3, and a transformer that maps country names to codes.

Explore data quality fundamentals, configure a data quality rule set on the existing blue catalog table, run the Glue job, evaluate outcomes, and enable CloudWatch alerts for failures.

Evaluate the quality of a CSV table in AWS Glue by applying a recommended data quality rule set, performing data quality checks, editing rules, and auditing results with CloudWatch metrics.

Create a glue job with inbuilt data quality checks using the evaluate data quality transformation on a CSV dataset, compare results, and halt on failures with a notification.

Set up a CloudWatch alarm for data quality checks in AWS glue, using a custom metric, evaluation context, and failure notifications via the SAS topic email, triggering on one-minute failures.