Ultimate AWS Data Engineering Bootcamp: 15 Real-World Labs

Explore batch data ingestion and orchestration with airflow using Python and Pandas, store data in S3, and load into Redshift serverless for reporting.

Discover how Redshift functions as a fully managed, serverless data warehouse in the cloud for large-scale analytics and complex queries.

Explore airflow as a managed AWS service, Moi, and learn to design data pipelines with DAGs, hooks, and operators, enabling scalable, cost-effective orchestration from extraction to ingestion.

Create an Amazon Redshift serverless workspace with a default namespace and work group, configure admin credentials and an IAM role, and enable public access and port 5439 for query execution.

Set up an AWS managed Airflow environment (not serverless) with version 2.9.2, creating an S3 bucket and folders for DAGs and requirements, uploading pandas, and configuring the VPC.

Develop and validate Python data transformations using pandas and Jupyter notebooks, inspecting songs, users, and streams csvs to produce genre and hourly kpi data frames for Airflow and Redshift deployment.

Explore an Airflow dag for key performance indicator calculations that validates datasets, branches on results, ingests from s3 to redshift via upsert, and archives processed files.

Create redshift tables for songs db and reporting schema with genre level KPIs and temp clones, configure airflow with S3 access, upload dags and Spotify data, and run the workflow.

Orchestrate data pipelines with airflow while transforming data in AWS Glue using PySpark and Python shells, load metrics into DynamoDB, and archive processed files.

Explore DynamoDB, a fully managed AWS NoSQL service with key-value and document models, delivering single-digit millisecond performance at scale for web apps, gaming, e-commerce, and real-time bidding.

Walk through running PySpark locally with AWS Glue docker image, test Spark code in a container, and generate top songs and genres using window functions before deploying to AWS Glue.

Deploy two spark and Python shell ETL jobs in AWS Glue by creating an IAM role with S3, DynamoDB, and CloudWatch permissions; spark writes to S3 while Python upserts DynamoDB.

Explore an end-to-end airflow dag that checks required files, triggers glue jobs for Spark and Python shell, archives input, and writes results to DynamoDB.

Learn to build cost-efficient AWS data pipelines for rental apartment analytics using Glue, Redshift, and Step Functions, from Aurora MySQL to raw, curated, and presentation layers in Redshift.

Discover aws aurora for mysql, a powerful relational engine delivering up to five times the throughput of standard mysql, with mysql compatibility, scale across three availability zones, and 99.99% availability.

Set up a MySQL aurora on AWS RDS with a dev test rental apartments DB, configure endpoints, connect via mysql, create tables (apartments, attributes, viewings), and load CSV data.

deploy and execute a mysql extraction glue job that reads from mysql tables and writes to s3 csvs, supporting incremental and full loads using secrets manager and dynamodb.

Create redshift raw tables cloned from source schemas, then deploy a star-schema process zone with a fact table and two dimensions to support upserts, and ingest via glue from s3.

Deploy a three-table Redshift ingestion pipeline with AWS Glue, using temp tables and incremental loads via last processed value, retrieving credentials from the secret manager to orchestrate the workflow.

Define and deploy a Step Functions state machine to orchestrate a multi-stage batch pipeline: extract MySQL data, ingest to Redshift raw, then process to the curated layer.

Set up a recurring AWS EventBridge schedule to trigger a Step Functions workflow using cron or rate expressions, configure retry and permissions, and monitor and clean up resources.

Build a data lake on AWS by processing rental marketplace data with spark on EMR, storing transformed outputs in S3, and querying with Athena via glue crawlers and step functions.

Explore AWS elastic map-reduce as a cloud native big data platform for processing data with Hadoop, spark, scoop, and Apache HBase; learn to set up EMR clusters and scale resources.

Explore how AWS Athena enables serverless, pay-as-you-go querying on S3 with no management overhead, supports compression and partitioning, and runs SQL with joins and window functions for data lake analysis.

Run two pyspark jobs locally in docker to compute rental metrics from csvs, producing transaction and user level data frames, and location and vehicle type insights, then deploy to EMR.

Launch and run spark jobs on an EMR cluster, configure EMR and EC2 roles, upload PySpark scripts to S3, and submit two steps to produce outputs and metrics.

Deploy four AWS Glue crawlers to infer schemas and populate the data catalog tables from S3 datasets; learn to configure IAM roles, run crawlers, and query with Athena.

Explore how to build a step functions workflow to spin up an emr cluster, run spark job one and two, and terminate the cluster, with roles and json definition.

Demonstrates running Spark jobs on EMR serverless, creating an EMR Studio and application, configuring a trust policy, submitting a batch job, and weighing costs against provisioned clusters.

Design an event-driven e-commerce data pipeline on AWS using ECS and step functions, where two Python containerized tasks validate S3 data and transform it into DynamoDB, triggered by EventBridge.

Explore the Elastic Container Service, a fully managed orchestration that simplifies building and scaling containerized apps, using Fargate or EC2, with task definitions and rolling updates.

Containerize and run a Python data validation app locally, validating datasets for required columns and data types, with Docker and S3-based file flows that move between new and ready folders.

Push the docker image to AWS elastic container registry, deploy to ECS with Fargate, create a task definition and IAM roles, run the task, and verify logs and data validation.

Transform data with AWS Wrangler and pandas-like operations across three datasets. Produce category wise and daily order summaries in DynamoDB with KPIs like revenue, average order value, and return rate.

Deploy and orchestrate two ECS tasks with a step function state machine, validating data, triggering on new S3 uploads. Update Docker image in ECR and revise task definitions.

Create an EventBridge rule with an event pattern to trigger a Step Functions state machine when an order items file is uploaded to the S3 bucket.

Build a lakehouse on AWS with Delta tables from raw e-commerce data using Spark, Glue Crawlers and Data Catalog on S3, and query with Athena.

Execute pyspark code to create delta lake locally with docker, processing products, orders, and order items datasets, performing upserts via delta tables using a delta jar and spark script.

Deploy PySpark on AWS Glue by configuring a Spark job, adding the Delta Lake jar from S3, and running for orders, items, and products before setting up crawlers and Athena.

Set up a Glue crawler to create Delta Lake tables in the data catalog and query them with Athena and Redshift, using a supermarket transactions database.

Learn to access delta tables from the Glue Data Catalog in Redshift, then use Athena to create a Delta Lake orders table for querying, with a daily automation option.

Explore event based data processing using AWS Kinesis streams for trip starts and ends, with Lambda functions updating a single DynamoDB table and Glue jobs producing aggregated insights to S3.

Explore AWS Kinesis for real-time data streaming and ingestion, with on-demand auto scaling, and learn to use Kinesis Data Streams and Kinesis Firehose alongside AWS Lambda and S3.

Set up Kinesis data streams for trip start and trip end, run a local Python script to read CSV files and write batches to the streams, then verify ingestion.

Deploy two lambda functions to listen to Kinesis streams, update a DynamoDB table, and trigger a Glue job that aggregates trips and writes the results to S3.

Learn how event-driven data processing works with AWS Kinesis and Lambda, tuning batch size and batch window. The lab also covers configuring Glue concurrency, and monitoring with CloudWatch and DynamoDB.

Transform real-time telecom data using Spark Streaming on AWS Glue, ingesting from Kinesis into S3, querying with Athena, and visualizing metrics in a Streamlit dashboard on ECS.

Walk through a PySpark streaming job from Kinesis to S3. Define a Spark schema with struct type, parse JSON, apply watermarking and two-minute windows, and write outputs every 20 seconds.

Deploy a spark streaming ETL job in AWS Glue that ingests data from the Kinesis data stream Mobile Coverage Logs and writes to S3.

Create three AWS Glue crawlers to catalog Spark streaming data in S3, producing partitioned tables in Mobile Network Aggregations for real-time SQL queries and a Streamlit app.

Learn to build a real-time streamlit dashboard, containerize the app with Docker, run locally on port 8501, and deploy to ECS using Athena queries and S3 outputs.

Deploy a Streamlit dashboard to AWS ECS as a containerized task by pushing to ECR, creating a Fargate cluster, and running the task to view data from Athena.

Learn how to implement CI/CD for data engineering with GitHub actions, automating builds, tests, and deployments via YAML workflows across dev, test, and prod environments.

Learn to deploy a lambda function and a glue job using AWS CLI, including zipping scripts and uploading to S3, with GitHub actions for CI/CD.

Deploy an AWS Glue python shell job via GitHub Actions to automate MySQL extraction with a workflow.yaml, and include unit tests for safe deployment.

Demonstrate end-to-end CI/CD for deploying an ECS task with GitHub Actions, dockerizing an app, pushing to ECR, and deploying via JSON task definition and workflow YAML.

Deploy a lambda function via GitHub actions CI/CD, configure a workflow yaml, install aws cli and boto3, and set up an IAM role with OpenID Connect for automated updates.