Data Engineering Bootcamp: SQL, ETL, PySpark & AWS(Hands-on)

Discover how AWS S3 powers a data lake, using buckets and prefixes, set file size limits up to five terabytes, and provide access methods and security for scalable cloud storage.

Navigate the AWS S3 console to create a globally unique bucket, explore prefixes as folders, upload a sample file, and learn how object URLs and URIs enable programmatic access.

Explore the three-layer data lake architecture—silver (raw data), gold (cleaned and transformed data), and data marts—showing how pipelines move data for analytics, business intelligence, and machine learning.

Explore data partitioning in data lakes to boost query performance and reduce costs, using date, geography, and service-type partitions plus best-practice partition pruning and maintenance.

Demonstrate data partitioning across silver, gold, and data warehouse layers; create S3 partitions and pipelines to transform raw data into curated analytics marts for BI.

Understand file formats in a data lake, from text and JSON to Parquet, Avro, and ORC. Compare data representation, compression, and schema, plus Delta Lake and Iceberg for analytics.

Master schema evolution to modify a data schema over time without reloading data, enabling backward and forward compatibility in data lake environments with Avro, Parquet, ORC, Delta Lake, and Iceberg.

Explore how the Hive metastore orchestrates metadata for data lakes, enabling tools like Trino, Presto, Spark, and AWS Glue Catalog to access partitioned data efficiently.

Explore how enabling versioning in an S3 bucket preserves object history, supports restoring previous versions, and applies retention policies for secure data lake management.

Explore AWS S3 storage classes to optimize cost, performance, and durability for data lakes and archives. Use lifecycle policies to automate transitions and manage retrieval, duration, and costs across classes.

Discover how AWS S3 events automate data lake workflows by triggering Lambda functions, notifications, and metadata updates, enabling real-time ingestion, transformation, and data quality checks.

Explore enabling s3 event notifications to trigger a lambda on file uploads, extract bucket and key from the event, and store file metadata for auditing.

Secure your data lake with AWS tools by implementing encryption, access control, and monitoring, using IAM, KMS, bucket policies, versioning, and cross-region replication to protect data integrity and compliance.

Explore how AWS S3 IAM and ACLs secure data lakes by controlling and auditing access with granular bucket, folder, and object permissions, policies, roles, encryption, and logging.

Master data lifecycle management in a data lake with AWS S3. Implement lifecycle policies, storage classes, and versioning to optimize cost, ensure compliance, and maintain accessibility.

Explore data lifecycle management by configuring S3 lifecycle rules for a data set prefix, transitioning objects to glacier after 30 days, with verification in the S3 console.

Explore backup and disaster recovery for an S3 data lake, covering cross-region replication, versioning, lifecycle policies, snapshots, and AWS Backup to ensure data resilience and business continuity.

Explore AWS S3 metadata to organize and search data in a data lake, using system defined metadata, user defined metadata, versioning, and event driven metadata updates for governance and ETL.

Explore AWS S3 Storage Lens, a comprehensive analytics tool that provides metrics and interactive dashboards to optimize storage cost, performance, and governance across accounts, regions, buckets, and prefixes.

Discover how to use the S3 storage lens dashboard to analyze data lake activity, monitor storage by bucket and prefix, view metrics, and plan for future data growth.

Explore data engineering concepts in data engineering bootcamp series 1 lab 7 by using boto3, the AWS SDK for Python, to list, upload, delete S3 objects and inspect S3 metadata.

Master data modeling for data lakes by defining entities, attributes, relationships, and schemas; apply normalization, oltp context, metadata management, and design considerations like schema on read, partitioning, and governance.

Explore the gold layer of a data lake and compare star schema, snowflake schema, one big table schema, and nested schema. Identify analytics and storage trade-offs for each model briefly.

Explore dimensional modeling and dimension tables, focusing on SCD1 and SCD2, surrogate and business keys, and how ETL preserves history for analytics.

Master fact modeling for data lakes, comparing star, snowflake, and nested schemas, and align fact tables with cost, scalability, security, partitioning, and query patterns.

Understand data marts as specialized subsets of a data warehouse that deliver fast, tailored insights for departments like payments. See how pre-aggregated, denormalized data supports dashboards, reporting, and data science projects.

Explore SCD type 1 in a two-layer data lake workflow, transforming raw drivers data into a dim drivers table with surrogate keys and audit timestamps.

Implement SCD type 2 in the dim drivers table to preserve history with surrogate keys and effective dates; use ETL to expire the current row and insert a new one.

Explore fact modeling and star-schema design by building raw and dimension tables (SCD2 and SCD1) and loading the fact table through ETL in a silver-to-gold data pipeline.

Drive trusted analytics by implementing data quality checks that ensure accuracy, completeness, reliability, consistency, and timeliness across data lake and data warehouse pipelines, protecting reports, costs, and reputation.

Explore types of data quality checks—completeness, accuracy, consistency, validity, uniqueness, and timeliness—and learn best practices like automating checks, establishing standards, and governance for data lake reliability.

Explore data quality tools that automate validation, monitoring, and remediation to ensure accurate data and trust in analytics, featuring Great Expectations, DK, Monte Carlo, and DBT.

Define and enforce data contracts between data producers and consumers to ensure a standardized structure, clear semantics, and trusted, high-quality data for analytics.

Explore data quality checks for dimensions and fact tables, including completeness, uniqueness, referential integrity, range, consistency, and format validations, with examples on dim drivers and fact bookings.

Compare PrestoDB, Trino, and AWS Athena to understand their strengths, deployment models, data source support, and serverless versus self-hosted approaches for big data analytics.

Explore creating external tables in AWS Athena using DDL, partitioned and non-partitioned data, register partitions with Glue Data Catalog, and navigate S3 storage for silver and gold data layers.

Query Athena with standard SQL to join bookings and location, compute trips and revenue by borough, create a monthly trip summary view, and explain workgroups for access control.

Explore using Athena via boto3 in Python to create DDL, register partitions, run queries with S3 results, and analyze logs for table popularity with a SQL parser.

Master AWS Athena cost and best practices by optimizing data formats (Parquet or ORC), partitioning, and compression, while monitoring costs with Cost Explorer, Budgets, and query logs for governance.

Explore Spark architecture, including the driver, executors, cluster managers, and Spark context, and see how in-memory, distributed processing, and shuffles enable scalable batch and real-time analytics.

Explore PySpark APIs, including Spark Context API and Spark Session API, to ingest, transform, query, and build scalable ML pipelines with data frames, SQL, and streaming.

Explore PySpark APIs through lab exercises, loading parquet and CSV data, creating views, and using data frame and SQL APIs to transform, join, and analyze taxi data.

Learn to implement a user defined function in PySpark to validate IP addresses. Register the UDF, apply it row by row, and add an is_valid column for data transformation.

Build and run a production grade data pipeline for taxi trips, applying transformations, enrichment, and schema validation, then load to staging and finally to the main path on aws emr.

Explains building a production-grade data pipeline with PySpark, transforming silver data to gold, applying data quality checks, schema validation, and staging-to-main path movement using S3.

Run a PySpark data pipeline on AWS EMR to load taxi trips data, transform it, and write to the gold layer with staging and date-based partitions and data quality checks.

Apply best practices for Spark and Parquet, including optimized cluster configuration, partitioning, and memory management. Use DataFrame API, avoid excess UDFs, and monitor with the Spark UI.

Explore airflow architecture by examining the components: scheduler, web server, metadata DB, workers, and executor, and how DAGs, tasks, and triggers coordinate scheduling and execution.

Set up airflow locally with Docker compose, configure Postgres and Redis, create DAGs, logs, and plugins folders, and access the Airflow UI after initial startup.

Design an hourly data ingestion pipeline in Airflow, building a dag with Python operators to fetch exchange rates from open exchange rates API, transform, and save parquet files to S3.

Explore Airflow DAG management, including pausing and activating DAGs, managing variables and connections, triggering runs, and ensuring idempotent data ingestion pipelines.

Develop an Airflow plugin that creates, submits PySpark jobs to, and terminates EMR clusters using Boto3, with DAG parameters and bootstrap actions guiding deployment.

Create an ETL data transformation pipeline with an Airflow DAG that provisions an EMR cluster, submits a PySpark job, and terminates the cluster, using cross-task communication and a reusable plugin.