Teach on Udemy

Turn what you know into an opportunity and reach millions around the world.

Learn More

Your cart is empty.

Keep shopping

PySpark Data Engineering: The Complete Real-World Workflow

Name: PySpark Data Engineering: The Complete Real-World Workflow
Rating: 4.4 (100 reviews)

Learn PySpark through a real project workflow: interactive dev, modular ETL, Dev/QA, Airflow, Git & production handoff.

Created byChandra Venkat

Last updated 8/2026

English

English [Auto],

What you'll learn

Understand Spark fundamentals and develop PySpark code interactively using Jupyter and PySpark Shell.
Turn interactive PySpark code into modular, reusable ETL applications using DataFrames and Spark SQL.
Run the same PySpark application across Dev and QA using environment-specific configurations and parameters.
Schedule and orchestrate PySpark pipelines using Airflow and manage code through real-world Git workflows.
Follow the complete project delivery workflow from development and testing to Jira-based production handoff.

Course content

8 sections • 38 lectures • 5h 45m total length

Why Spark Solves Real ETL Problems2:02
Learn why PySpark is preferred for large-scale data pipelines over traditional tools.
Spark’s Role in Data Pipelines2:20
See where Spark fits in modern data workflows, from raw to processed data
Spark Jobs, Stages, DAGs — Quick Intro4:12
Get a fast overview of how Spark executes jobs internally

What You’ll Set Up – Spark + Airflow + HDFS in Docker1:42
Understand the tools you’ll set up and how they work together.
Install WSL2 + Docker5:18
Quickly set up Docker on your machine, even on Windows using WSL2.
Create Docker Compose Setup2:54
Build your complete data engineering environment with a single Compose file
One Command: Launch Spark, Airflow, HDFS, Jupyter12:55
Run your entire stack with one command like a real dev.

Important (Read Before Running Docker)
If you see this error while running docker compose up:
“permission denied while connecting to Docker daemon”
or
“docker.sock: connect: permission denied”

Run this once inside Ubuntu:
sudo usermod -aG docker $USER

Then close Ubuntu completely and open it again.
After this, docker compose up will work normally.
Docker Compose Setup - Troubleshooting Guide10:32
Troubleshoot docker compose up failures with a clean reset, ensure docker desktop is running, handle spark and hdfs download issues manually, fix port conflicts, and adjust docker resources on Mac.
Explore Spark UI, Airflow UI, HDFS UI5:59
Learn how to navigate the UIs you’ll use to monitor jobs and data
Explore HDFS10:27
Upload datasets into HDFS for real-world big data pipelines.
Use PySpark Shell4:32
Start your first interaction with PySpark and explore basic commands

Load & Explore Files in Jupyter and PySpark Shell14:41
Load and inspect datasets interactively using Jupyter + PySpark.
Build ETL Logic Using DataFrame API (Joins, Transforms, Write)17:43
Create ETL pipelines using PySpark’s powerful Spark DataFrame API.
Perform Same ETL in Spark SQL11:26
Create ETL pipelines using PySpark’s Spark SQL
Write Clean PySpark Scripts9:12
Organize your ETL logic into reusable, clean scripts for production.
Submit Jobs Using spark-submit3:42
Run PySpark jobs in cluster mode using spark-submit.
(Optional)How to Navigate Spark-apps10:57
Navigate the spark-apps workspace, locate the pi spark stack on the C drive, access spark apps, notebooks, and DAGs folders, and manage docker compose up and down from ubuntu.

Understand Sales ETL Requirement10:47
Understand business requirements behind real-world sales data pipelines
Build the Sales ETL Pipeline Using Pyspark Shell20:43
Develop a full ETL pipeline for sales data using PySpark.
Create a Reusable PySpark Script11:02
Package your ETL logic into a modular, team-friendly script.
Dynamic Shell Script for Deployment11:20
Create a dynamic shell script to deploy ETL jobs easily.
Schedule Daily Run with Cron (Simulate Lightweight Prod)9:55
Set up a daily job scheduler with cron, simulating production workflows.
Wrap-Up and Key Learnings13:19
Summarize what you built and how it fits into real production setups.
Notes/Commands used in Sales ETL Pipeline3:14

Understand Customer ETL Requirement4:01
Understand the architecture behind customer data pipelines.
Build Pipeline using Jupyter notebook22:29
Build a production-ready customer ETL job with PySpark.
Convert Workflow into Pipeline Script11:21
Convert a jupyter notebook into a .py script with nbconvert, package it in the spark apps folder, run spark submit, and verify hdfs output for airflow scheduling.
Create Shell Wrapper script13:06
Set up wrapper scripts to automate job deployments.
Automate Pipeline with Airflow22:15
Design, schedule, and run your pipeline using Airflow DAGs.
Wrap-Up and Lessons Learned4:54
Review the pipeline and understand real-world deployment lessons.

Why Dev vs Prod Matters – Company Best Practices3:11
Learn the importance of separating development, QA, and production environments
Create Env shell to make Pipeline Dynamic3:17
Create dynamic shell scripts to switch environments easily.
Switch Between Dev and Prod25:36
Practice switching modes without changing your codebase.
Your Production-Ready Pipeline Setup4:24
Build a flexible setup that real companies use for pipeline management.

Requirements

Basic Python knowledge
Familiarity with SQL is helpful but not mandatory.
No prior experience with Spark, Docker, or Airflow is required; everything is taught step-by-step
A computer with at least 8 GB RAM (12 GB recommended) and 40 GB free disk space (50 GB recommended)
A good internet connection

Description

Want to learn PySpark for Data Engineering and understand how it is actually used in a real project?

This course goes beyond isolated PySpark syntax and transformations to show you the complete workflow around PySpark code in a Data Engineering project.

You'll see how PySpark code is developed interactively, turned into modular ETL applications, run across Dev/QA environments, orchestrated with Airflow, managed through Git, monitored using Spark UI and logs, and finally taken through the Production handoff process.

What You'll Learn

Develop PySpark interactively using Jupyter and PySpark Shell
Build ETL pipelines using DataFrames and Spark SQL
Turn interactive code into modular, reusable PySpark applications
Structure applications using scripts, configs, environment files, and reusable modules
Run applications using spark-submit
Run the same application across Dev and QA environments
Schedule and orchestrate pipelines using cron and Airflow
Manage code using Git branching and merging workflows
Monitor and troubleshoot applications using Spark UI and logs
Follow the Production handoff and deployment process

Hands-On Project Workflow

You'll work with Spark/PySpark, Airflow, Docker, HDFS, Jupyter, and Git while building end-to-end Data Engineering projects.

The course follows the journey:

Interactive Development → ETL → Modular Application → Dev/QA → Orchestration → Git → Monitoring → Production Handoff

Who Is This For?

This course is for Data Engineers, developers, and ETL professionals who want practical PySpark experience and want to understand how PySpark code fits into the broader Data Engineering project workflow.

By the End

You won't just know how to write PySpark code. You'll understand how that code is developed, structured, executed, orchestrated, managed across environments, monitored, and taken through the Production handoff in a Data Engineering project.

Who this course is for:

Aspiring data engineers who want hands-on, realistic project experience.
Python developers or analysts transitioning into data engineering roles.
Students and self-learners seeking portfolio-worthy PySpark projects.
Professionals preparing for real-world Spark-based roles and interviews.

PySpark Data Engineering: The Complete Real-World Workflow

What you'll learn

Explore related topics

Course content

Start Your Data Engineering Journey3 lectures • 9min

Set Up Your Data Engineering Stack8 lectures • 54min

Build and Test PySpark ETL Jobs6 lectures • 1hr 8min

Real Project: Sales ETL Pipeline (Cron)7 lectures • 1hr 20min

Real Project: Customer ETL Pipeline (Airflow & HDFS)6 lectures • 1hr 18min

Production Ready: Dev vs Prod Pipelines4 lectures • 36min

Workflows — Git, Handoffs, Deployments2 lectures • 19min

Thank you!!2 lectures • 2min

Requirements

Description

Who this course is for: