Python, SQL & Spark for Data Engineering and Adv Terraform

Master Python foundations for data engineering, covering data structures, file handling, error handling, logging, debugging, and production-grade practices, while showing how Python connects systems and prepares data for analytics.

Explore terraform basic blocks, including the terraform block, provider block, and resource block, and learn how versions.tf and root modules organize providers, resources, and provisioners for a real world model.

Set up a data engineering project by creating a virtual environment, activating it, and listing dependencies in a requirements.txt. Organize data and src directories with readme.md and bronze-silver-gold layers.

Explore how AWS CloudWatch Container Insights collects metrics and logs from containerized apps on EKS and ECS, uses a containerized CloudWatch agent, and enables alarms, dashboards, and log groups.

Implement context path based routing with ingress rules to route /app1 to app1 and /app2 to app2 node port services, with default handling via the ingress and terraform automation.

Learn how to deploy multiple ingress services across namespaces to create a single AWS application load balancer using ingress groups, enabling cross-namespace load balancing and cost efficiency.

Explore how Terraform modules package and reuse resource configurations as root and child modules, loaded from local or public and private registries, including building and publishing a local module.

Master Terraform by automating Kubernetes YAML deployments with the Cube CTL provider, using HTTP data sources and kubectl manifest to deploy CloudWatch agent and Fluentbit on EKS.

Explore Terraform remote state storage and state locking using AWS S3 and DynamoDB, and apply these backends to multi-project deployments like EKS clusters and Kubernetes resources.

Provision EKS admins and read-only users with AWS IAM roles and groups, update AWS IAM auth configmap, and implement Kubernetes cluster roles and bindings, using both manual and Terraform automation.

Explore string, numeric, and date sql functions, including lower, upper, concat, substring, length, replace, round, abs, power, random, and date methods, plus inline, correlated, and derived subqueries.

Explain how Spark lazy evaluation enables global optimization and efficient failure recovery. Recomputes only the required data, handles multiple actions, and may reorder to reduce shuffles.

Compare shuffle join and broadcast join in Spark, highlighting when to broadcast a small table, the cost of shuffles, and how partitioning, repartitioning, and coalesce affect data balance and performance.

Explore Snowflake as a modern cloud data warehouse, covering topics from beginner to advanced with live demonstrations, and learn how Snowflake leases storage and compute from cloud providers.

Explore data ingestion and power query in Power BI by importing CSV, Parquet, or JSON, connecting to Postgres, and transforming data types, removing nulls, and viewing SQL and M language.

Learn how to use dbt for ELT transformations in a dockerized production setup, orchestrating SQL transformations in a Postgres warehouse via a DAG, with Git and CI practices.