Python, SQL & Spark for Data Engineering Hands-On Course

Explore how databases organize data and compare relational SQL databases with NoSQL systems. See why PostgreSQL is chosen, with hands-on demonstrations and set checkpoints to reinforce concepts.

learn to install postgresql and dbeaver, connect them, create an e-commerce database and schema, and write and execute basic sql queries with proper semicolon endings and readability rules.

Learn how to design sql tables in the ecommerce schema, create a customers table with serial id, varchar fields, constraints like primary key and unique email, and default timestamps.

Explore acid properties of transactions, data consistency, isolation, and durability, while practicing crud and dcl commands with hands-on demonstrations of begin, commit, rollback, and savepoint in a relational database.

Master select queries to retrieve, filter, and sort data using where, operators, and patterns. Apply order by, limit, fetch, and aggregate functions with group by and having.

Learn how order by, limit and fetch control result sets, including offset behavior. Explore aggregate functions, group by and having, and null handling with coalesce, null if, and case logic.

Explore string, numeric, and date functions in SQL, including lower, upper, concat, substring, length, and age. See current date, extract, now, date truncate, and inline, correlated, and derived subqueries demonstrated.

Explore inline, correlated, and derived subqueries, using table aliases, joins, set operators, common table expressions, and window functions to analyze data.

Learn how Apache Spark enables distributed data processing across multiple machines with PySpark, a Python API, offering a high-level programming model for scalable, big data workflows.

Learn Apache Spark architecture with a driver and multiple executors, task scheduling, the catalyst optimizer turning logical plans into physical plans, and a MapReduce comparison.

Debug a Spark master, manage executors, and inspect jobs, stages, and tasks in the Spark UI. Learn lazy evaluation: transformations run only after an action to optimize and avoid work.

Discover how Spark uses lazy evaluation and global optimization to reorder transformations, reduce shuffles, and recompute only what's needed for efficient failure recovery.

Explore how Spark reads and writes CSV data, demonstrates distributed partitioning, and compares CSV, JSON, and Parquet formats, including schema inference, compression, and column pruning.

Explore how Spark writes and reads CSV, JSON, and Parquet; compare schema inference versus explicit schema, and see Parquet's speed and compactness for big data.

Define and read data with struct and DDL schemas, distinguish narrow and wide transformations and their shuffles, and master basic transformations (select, filter, withColumn) while optimizing expression order.

Explore predicate pushdown and early filtering in spark to reduce computation and data transfer, then examine lazy evaluation, actions, catalyst optimizer, and the two join types: parallel and broadcast.

Explore Spark window functions such as ranking, running totals and moving averages, and see how partition by and order by along with repartition and coalesce influence skew and performance.

Demonstrate managed versus external tables in Spark, their metastore storage, data location, and drop table effects, and explain Spark SQL execution model and declarative versus programmatic queries.

Connect Snowflake to Google Cloud Storage as an external stage, configure storage and notification integrations, set up PubSub, and deploy Snowpipe to load data into a raw table.

Discover Apache Airflow as a workflow orchestrator using dag-based tasks and operators to automate pipelines, with monitoring and ci/cd support, while noting it is not an etl tool or processor.

Explore the architecture of Apache Airflow and navigate the Airflow UI to design and monitor data pipelines, within a hands-on data engineering course that covers Python, SQL, and Spark.

Explore Apache Airflow operators and categories in a hands-on data engineering context, and see a Bash operator demo.

Master Apache Airflow scheduling syntax to manage data pipelines, and learn how backfill and catchup control task execution in data engineering workflows.

Explore Apache Airflow task groups through a hands-on demo within the Python, SQL & Spark for data engineering course. Learn how task groups streamline workflows in data engineering projects.

Explore building a bank data pipeline using sql, PostgreSQL, kafka, debezium for change data capture, minio S3 storage, snowflake with dbt, dockerized airflow, and Power BI direct query.

Design a banking data model in PostgreSQL by creating accounts, customers, and transactions tables with keys, foreign keys, and constraints, illustrating snowflake schema connections and data generation.

Create a Postgres database and build a modern data pipeline from ingestion to visualization. Capture changes with Debezium and Kafka, store in S3 and Snowflake, and orchestrate with Airflow.