Name: Data Engineering Vol2 AWS : Data Processing - Spark & Kafka
Rating: 4.9 (14 reviews)

Udemy Business

Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

Created bySoumyadeep Dey

Last updated 2/2026

English

What you'll learn

Deep dive on Spark and Kafka using AWS EMR, Databricks, MSK
Understand Data Engineering (Volume 2) on AWS using Spark and Kafka
Batch and Stream processing using Spark and Kafka
Production level projects and hands-on to help candidates provide on-job-like training
Get access to datasets of size 100 GB - 200 GB and practice using the same
Learn Python for Data Engineering with HANDS-ON (Functions, Arguments, OOP (class, object, self), Modules, Packages, Multithreading, file handling etc.
Learn SQL for Data Engineering with HANDS-ON (Database objects, CASE, Window Functions, CTE, CTAS, MERGE, Materialized View etc.)
AWS Data Analytics services - S3, EMR, Databricks, MSK

Course content

20 sections • 193 lectures • 47h 25m total length

Introduction - Data, Data Lifecycle & Data Engineering Pipeline26:28
Data Engineering Volume 2 Course & Projects Overview, Roles in Data25:18
AWS Resource Cost for the Course6:25

Spark Cluster on AWS EMR21:53
Spark Standalone Cluster on Laptop & EC2, Spark UI29:23
Spark Application Deployment Modes (--deploy-mode cluster|client)3:11
DataFrames, DAG, PySpark Module24:22
SparkSession20:16
DataFrame Reader & Writer28:39
Define DataFrame Schema - StructType, StructField, inferSchema21:05
Partition, Split, Task, Executor Relation17:49
Partitions & Splits for Smaller Files17:37

Introduction to Transformation15:58
HandsOn : Intro to Transformation17:58
Step-by-step Transformation18:26
Chain Transformation5:46
Transformation - describe, sort, limit, drop, dropDuplicates19:15
Spark Action & Lazy Loading8:36
HandsOn : Spark Action & Lazy Loading15:12
HandsOn : Spark UI - Jobs, Executors, Environment10:04
HandsOn : load() - Transformation or Action??13:23
HandsOn : File Performance Comparison - CSV vs Parquet11:36
Assignment Project 1 - Transformation & Action10:47
Transformation - agg, groupBy, orderBy, selectExpr, subtract, withColumn, union7:57
HandsOn : Transformation : agg(), alias(), groupby()14:07
HandsOn : Transformation : join()18:59
HandsOn : Function : col()19:20
HandsOn : Transformation : orderBy(), selectExpr()9:13
HandsOn : Transformation : subtract()9:02
HandsOn : Transformation : union(), unionAll(), intersect(), withColumn()10:09
Shuffle Process & External Shuffle Service (ESS)23:52
HandsOn : Shuffle Process in groupBy() - Explanation using Spark UI24:12
What should be the value of Shuffle Partition No (spark.sql.shuffle.partitions)13:21
Assignment Project 2 : Shuffle & Spark UI19:59

Do Dataframes keep occupying memory ??9:39
How Spark Applications use Memory CPU & its calculation12:53
How to decide the size of Spark PROD Cluster14:09
Explore Spark UI - WholeStageCodegen, MapPartitions, Exchange, etc.22:08
Memory Management - Reserve, Execution, Storage & User Memory14:44
Memory Spilling, Unified Memory Management & Off-heap Memory11:04
Cache, Persist, Unpersist, ClearCache7:04
HANDS-ON : Monitor Memory Spill, Caching & Persist in Spark UI13:54
Garbage Collection and Kryo Serialization in Spark10:20

Dataframe collect() API10:35
foreach() & foreachPartition() to apply customer logic on dataframe records16:50
Loop/Iterate though Dataframe records - toLocalIterator(), df.na.fill()16:52
Spark Functions - lit(), concat(), expr()13:15
DATE Functions15:22
when() & otherwise() Functions6:25
WINDOW Functions9:23
HandsOn : WINDOW Functions7:32
Assignment : Spark WINDOW Functions18:08
Assignment Project 4 - Layover Analysis, Travel Activity, Environment Impact36:28
lead(), lag(), nth_value(), first_value(), last_value()11:02
HandsOn : lead(), lag(), nth_value(), first_value(), last_value()6:49

Spark Catalyst Optimizer & Tungsten Execution Engine16:57
Spark Internal Joins - Sort Merge Join (SMJ)15:11
Spark Internal Join - Broadcast Hash Join (BHJ)9:56
Spark Internal Join - Broadcast Nested Loop Join (BNLJ)6:06
Spark Internal Join - Shuffled Hash Join (SHJ)5:55
Spark Internal Join - Shuffled NL Join (SNLJ), Join Hints, Join Performance5:11
SparkUI Operators - HashAggregate, SortAggregate, BroadCastExchange12:34
SparkUI Operators - BroadcastHashJoin, BroadcastNLJoin5:58
SparkUI Operators - ShuffledHashJoin, Limit, CollectLimit, Window, ReusedExchang7:39
HANDS-ON : Spark Operators7:28
Spark Explain & How to Read Explain Output21:06
Data Skewness or Skewed Data8:10
HANDS-ON : Analyse & Find Data Skewness using Spark UI14:51
HANDS-ON : Spark Salting - Mitigate Data Skewness and Skew Joins9:12
HANDS-ON : Implement Salting step-by-step16:55
Adaptive Query Execution (AQE), AQE Parameters17:40
HandsOn : AQE Coalesce Shuffle Partitions9:41
HANDS-ON : AQE Optimize Data Skewness & Skewed Join4:24
Dynamic Partition Pruning7:24
Dynamic Resource Allocation (DRA)17:40
SUPER SUMMARY10:29

EMR Hosting Platform Introduction23:19
EMR Cluster Creation with IAM Roles13:45
EMR Considerartions - Spark, YARN Configuration Parameters, Submit Spark code to17:39
HandsOn : Prepare EMR environment to execute Spark Applications12:02
HandsOn : Submit Spark application to EMR from Primary Node & Laptop20:02
HandsOn : Run Multiple Spark Applications one-by-one using EMR Steps9:41
Jupyter Notebook on EMR - Introduction9:32
HandsOn : Jupyter Notebook Security Group & IAM pre-requisites6:46
HandsOn : Create Jupyter Notebook and execute PySpark application19:47
Relational (RDBMS) Database Sources & Targets15:22
HANDS-ON : Write to RDBMS (MySQL) Target22:26
HANDS-ON : Read from RDBMS (MySQL) Source5:31
Data Warehouse Source & Targets - Redshift13:19
HANDS-ON : Read & Write from/to Redshift Data Warehouse16:39

Requirements

Good to have AWS and SQL knowledge

Description

This is Volume 2 of Data Engineering course. In this course I will talk about Open Source Data Processing technologies - Spark and Kafka, which are the most used and most popular data processing frameworks for Batch & Stream Processing. In this course you will learn Spark from Level 100 to Level 400 with real-life hands on and projects. I will also introduce you to Data Lake on AWS (that is S3) & Data Lakehouse using Apache Iceberg.

I will use AWS as the hosting platform and talk about AWS Services - EMR, S3 and MSK. I will cover Databricks as Spark hosting platform. I will also show you Spark integration with other services like AWS RDS (MySQL or PostgreSQL) and Redshift.

You will get opportunities to do hands-on using large datasets (100 GB - 300 GB or more of data). This course will provide you hands-on exercises that match with real-time scenarios like Spark batch processing, stream processing, performance tuning, streaming ingestion, Window functions, ACID transactions on Iceberg etc.

Some other highlights:

10 Projects with different datasets. Total dataset size of 250 GB or more.
Other technologies covered - EC2, EBS, VPC and IAM.
Optional Python videos
Optional AWS and SQL Essentials videos

I will conclude the Data Engineering course with Volume 3, in which, I will be covering the following Topics.

Flink
Apache Airflow
Apache Pinot
AWS Kinesis

Please provide feedback and suggestions if you want me to add any other topics.

Who this course is for:

Python developers, Application Developers, Big Data Developers
Data Engineers, Data Scientists, Data Analysts
Database Administrators, Big Data Administrators
Data Engineering Aspirants
Solutions Architect, Cloud Architect, Big Data Architect
Technical Managers, Engineering Managers, Project Managers

What you'll learn

Explore related topics

Course content

Introduction to Data Engineering Volume 23 lectures • 58min

Big Data Processing6 lectures • 2hr 29min

Introduction to Spark3 lectures • 1hr 25min

Knowing Spark - Up Close Part 19 lectures • 3hr 4min

Spark Transformation & Action - Part 122 lectures • 5hr 17min

Spark Partitions - Input, Shuffle & Output4 lectures • 50min

Knowing Spark - Up Close Part 29 lectures • 1hr 56min

Transformation & Action Part 2 + Spark Functions12 lectures • 2hr 49min

Knowing Spark - Up Close Part 321 lectures • 3hr 50min

Hosting Platforms - AWS EMR (Elastic MapReduce)14 lectures • 3hr 26min

Requirements

Description

Who this course is for: