Python, SQL & Spark for Data Engineering and Adv Terraform

Master Python foundations for data engineering, covering data structures, file handling, error handling, logging, debugging, and production-grade practices, while showing how Python connects systems and prepares data for analytics.

Explore how Python drives data engineering by handling extraction, ingestion, validation, transformation, and orchestration across data sources like APIs, files, Kafka, and databases, enabling analytics with SQL and Spark.

Set up a data engineering project by creating a virtual environment, activating it, and listing dependencies in a requirements.txt. Organize data and src directories with readme.md and bronze-silver-gold layers.

Explore how AWS CloudWatch Container Insights collects metrics and logs from containerized apps on EKS and ECS, uses a containerized CloudWatch agent, and enables alarms, dashboards, and log groups.

Explore Python data structures including lists, tuples, sets, and dictionaries, covering mutability, ordering, duplicates, indexing, and practical use in data engineering.

Learn how to deploy multiple ingress services across namespaces to create a single AWS application load balancer using ingress groups, enabling cross-namespace load balancing and cost efficiency.

Explore how Terraform modules package and reuse resource configurations as root and child modules, loaded from local or public and private registries, including building and publishing a local module.

Master Terraform by automating Kubernetes YAML deployments with the Cube CTL provider, using HTTP data sources and kubectl manifest to deploy CloudWatch agent and Fluentbit on EKS.

Explore Terraform remote state storage and state locking using AWS S3 and DynamoDB, and apply these backends to multi-project deployments like EKS clusters and Kubernetes resources.

Provision EKS admins and read-only users with AWS IAM roles and groups, update AWS IAM auth configmap, and implement Kubernetes cluster roles and bindings, using both manual and Terraform automation.

Learn how to implement exception flow in Python by using try, except, and finally to handle runtime errors, prevent pipeline breaks, and adapt to multiple possible outcomes.

Create a Snowflake demo that builds an orders table, inserts data with a Python driver, and selects rows, highlighting cursor and connection management and Airflow automation, plus API basics.

Design an e-commerce database by creating tables for customers, orders, products, and categories; define primary and foreign keys, one-to-many relations, and on delete cascade using create table queries.

Learn how to write effective select queries to filter, sort, and aggregate data using where, between, like, in, and/or, plus group by and having; master order by, limit, fetch.

Explore string, numeric, and date sql functions, including lower, upper, concat, substring, length, replace, round, abs, power, random, and date methods, plus inline, correlated, and derived subqueries.

Explore window functions such as row_number, rank, dense_rank, lag, lead, ntile, and aggregate sums, averages, and counts over partitions; compare views and materialized views, plus basic procedures.

Explain how Spark lazy evaluation enables global optimization and efficient failure recovery. Recomputes only the required data, handles multiple actions, and may reorder to reduce shuffles.

Compare shuffle join and broadcast join in Spark, highlighting when to broadcast a small table, the cost of shuffles, and how partitioning, repartitioning, and coalesce affect data balance and performance.

Discover managed and external tables, Hive metastore concepts, Spark SQL vs DataFrame execution, and Spark streaming basics in a practical data engineering course.

Explore Snowflake as a modern cloud data warehouse, covering topics from beginner to advanced with live demonstrations, and learn how Snowflake leases storage and compute from cloud providers.

Learn how to set up a Snowflake project with streams for change data capture, merge into a target table, and schedule a sales merge task.

Explore Apache Airflow operators and categories, and demo the Bash operator to demonstrate scheduling tasks in data engineering workflows.

Explore data ingestion and power query in Power BI by importing CSV, Parquet, or JSON, connecting to Postgres, and transforming data types, removing nulls, and viewing SQL and M language.

Learn how to use dbt for ELT transformations in a dockerized production setup, orchestrating SQL transformations in a Postgres warehouse via a DAG, with Git and CI practices.

Explore how dbt builds and inspects a dag from staging to fact tables using ref and source, and manage upstream and downstream dependencies with view, table, and incremental materialization.

Update a stg orders model with dbt run and snapshot, illustrating slowly changing dimensions, and introduce custom tests using yaml generic tests and sql singular tests.

Explain text execution strategy and CI enforcement for data pipelines, using dbt source freshness, run, and tests, and show how modified parts and manifest.json drive targeted validation before deployment.

Create yaml documentation specs describing models and their columns, with docs living beside the models. Generate artifacts with dbt docs to build manifest.json, catalog.json, and an interactive site.