Apache Spark for Big Data with Python and PySpark

Master Apache Spark, PySpark, Spark SQL, DataFrames & Big Data Analytics

New

Created byTutorac Inc

Last updated 7/2026

English

What you'll learn

Understand Apache Spark architecture and distributed computing concepts.
Build Big Data applications using PySpark and Python.
Work with RDDs, DataFrames, and Spark SQL for large-scale data processing.
Perform Spark transformations, actions, and data analytics efficiently.
Configure Spark environments and AWS for Big Data workloads.

Course content

15 sections • 72 lectures • 18h 39m total length

Understanding Apache Spark10:15
Get introduced to Apache Spark and understand its purpose, architecture, and role in modern big data processing.
Realtime use cases of Spark Applications10:03
Explore real-world Spark applications and understand how Spark is used for analytics, machine learning, streaming, and large-scale data processing.
Understanding Advantages over HADOOP Frame Work5:19
Learn the advantages of Apache Spark over the Hadoop framework and understand why Spark delivers faster data processing.
Apache Spark Framework Components16:39
Understand the major components of the Apache Spark framework and how they work together to process big data efficiently.

Overview of Spark Architecture7:12
Learn the overall architecture of Apache Spark and understand how distributed data processing is organized.
Understanding Driver Process, Executors & Cluster Manager5:51
Explore the roles of the Driver, Executors, and Cluster Manager in executing Spark applications.
Understanding Correlation between Architecture & Process6:40
Understand how Spark architecture and execution processes work together to perform distributed computing.
What are the Primary Components of Spark Architecture2:30
Learn about the primary components that form the foundation of Apache Spark architecture.
How is Spark Different from HADOOP2:41
Compare Apache Spark and Hadoop to understand their differences, strengths, and ideal use cases.

Understanding Key Concepts2:00
Understand the core concepts of Spark Structured APIs and their role in distributed data processing.
The Spark Session4:00
Learn about SparkSession and understand how it serves as the entry point for Spark applications.
DataFrames6:46
Learn how DataFrames organize structured data and simplify data processing in Apache Spark.
Partitions6:49
Understand Spark partitions and learn how they improve parallel data processing performance.
Transformations17:12
Learn how Spark transformations create new datasets through lazy execution.
Lazy Evaluation11:25
Understand the concept of lazy evaluation and how Spark optimizes execution plans.
Actions7:34
Learn how Spark actions trigger execution and return results from distributed datasets.
An End to End Example21:47
Follow a complete Spark workflow from reading data to processing and generating results.
Overview of Structured API Execution5:59
Understand how Spark Structured APIs execute queries and optimize workloads.
Understanding Logical & Physical Planning Process8:26
Learn how Spark builds logical and physical execution plans for efficient query processing.
Create Spark Session2:59
Learn how to create and configure a SparkSession for Spark applications.
Difference Between Actions & Transformations2:14
Understand the differences between Spark actions and transformations with practical examples.
What are the Different Stages During Spark Job Execution3:25
Learn the various stages involved in Spark job execution and how tasks are distributed.
How Wide & Narrow Transformations Differ2:37
Understand the differences between wide and narrow transformations and their impact on Spark performance.

Data Types6:48
Learn about Spark data types and how they are used in structured data processing.
Columns & Expressions21:55
Understand how columns and expressions are used to manipulate structured datasets.
Records & Rows15:16
Learn how Spark represents data using records and rows within DataFrames.
Dataframe Transformations & Operations39:03
Explore common DataFrame transformations and operations used for large-scale data analysis.
Reading from Different Sources10:40
Learn how to load data into Spark from various data sources and file formats.
Other Basic Structured Operations25:57
Explore additional structured operations available for processing Spark DataFrames.
Excercises for Structured Operations10:00
Practice structured data operations through hands-on Spark exercises.

Aggregation Functions15:13
Learn how aggregation functions summarize large datasets by calculating statistical and analytical values.
Grouping Functions8:31
Understand grouping functions and learn how to organize and summarize data efficiently.
Window Functions23:59
Learn how window functions perform calculations across related rows without changing the dataset structure.
Joins25:43
Understand different types of joins and learn how to combine data from multiple datasets.
Aggregation Excercise6:30
Apply aggregation concepts through practical exercises using Spark DataFrames.

Understanding the Low Level APIs9:20
Learn about Spark's low-level APIs and understand how RDDs provide fine-grained control over distributed data processing.
Creating RDDS18:24
Learn how to create Resilient Distributed Datasets (RDDs) in Apache Spark.
Data Transformation with RDDS24:06
Perform data transformations using RDD operations for scalable distributed computing.
RDDS Excercise8:16
Practice creating and transforming RDDs through hands-on Spark exercises.

Understanding Deployment Modes14:42
Learn the different deployment modes available for running Spark applications across various cluster managers.
LifeCycle of Spark Applications8:47
Understand the complete lifecycle of a Spark application from submission to execution and completion.
Understanding Jobs, Stages & Tasks42:47
Learn how Spark divides workloads into jobs, stages, and tasks for distributed execution.
Pipelining5:05
Understand Spark pipelining and learn how it optimizes execution for better performance.
Deploying Spark Applications5:21
Learn how to package, configure, and deploy Apache Spark applications in production environments.

Requirements

Basic Python programming knowledge is recommended.
Basic understanding of SQL is helpful but not required.
A computer with internet access.
No previous Apache Spark experience is required.

Description

"Py - Spark" is a specialized course designed for individuals looking to harness the power of Apache Spark with PySpark. Apache Spark is a unified analytics engine for big data processing, while PySpark provides an easy-to-use interface for Python developers to leverage Spark's capabilities. This course will guide you through the fundamentals of PySpark, from setting up your environment to performing complex data transformations and analysis at scale.

Course Highlights:

Apache Spark & PySpark API: Learn about RDDs and DataFrames.
Data Manipulation: Explore how to manipulate large datasets using PySpark's powerful transformations and actions.
Real-World Applications: Work on hands-on projects and exercises that simulate real-world data scenarios.

You will start by learning the basics of Apache Spark and the PySpark API, including RDDs and DataFrames. These are core to Spark's data processing capabilities. You'll explore how to manipulate large datasets using PySpark's powerful transformations and actions, gaining proficiency in handling distributed computing tasks.

Advanced Topics:

Data Cleaning and Preprocessing: Learn essential techniques for preparing your data.
SQL Queries with Spark SQL: Perform complex SQL queries efficiently.
Optimizing Spark Jobs: Techniques to enhance performance and efficiency.

Throughout the course, you'll work on hands-on projects and exercises that simulate real-world data scenarios, allowing you to apply your knowledge. Topics covered include data cleaning and preprocessing, performing SQL queries with Spark SQL, and optimizing Spark jobs for performance.

By the end of this course, you'll be equipped with the skills to utilize PySpark effectively for big data analytics and processing tasks. Whether you're a data scientist, data engineer, or developer aiming to work with large-scale datasets, "Py - Spark" will empower you to leverage Apache Spark's capabilities through Python.

Join us and unlock the potential of PySpark for your big data projects!

Who this course is for:

Python developers interested in Big Data.
Data Engineers and aspiring Spark Developers.
Data Analysts working with large datasets.
Software Developers learning distributed data processing.

Apache Spark for Big Data with Python and PySpark

What you'll learn

Explore related topics

Course content

Spark Overview4 lectures • 42min

Apache Spark Architecture5 lectures • 25min

AWS Basic Setup2 lectures • 33min

Structured APIs - Data Frames, SQL, Datasets14 lectures • 1hr 43min

Structured Operations7 lectures • 2hr 10min

Aggregations5 lectures • 1hr 20min

Spark SQL2 lectures • 37min

RDDs4 lectures • 1hr

Deploying Spark Applications5 lectures • 1hr 17min

Monitoring & Debugging2 lectures • 47min

Requirements

Description

Who this course is for: