Learn Big Data Analysis with PySpark

Name: Learn Big Data Analysis with PySpark
Rating: 4.4 (27 reviews)

Learn Big Data Analysis in PySpark using the Apache Spark's Powerful Features and Easy Commands of Python and SQL

Created byData Science Guide

Last updated 8/2025

English

What you'll learn

Learn Most Important PySpark Features
Understand Resilient Distributed Dataset
Learn Most Important Python Commands and Libraries used for Data Analysis
Import Big Data Files in PySpark Work Environment and Clean them
Perform Data Analysis in PySpark using SQL Queries

Course content

4 sections • 38 lectures • 1h 55m total length

Introduction to the Course2:28
What is PySpark5:05
What is PySpark0:14
Spark Features2:53
Resilient Distributed Dataset (RDD)4:30

Introduction to development platform (Colaboratory)2:08
Login to Colaboratory1:19
Create Lists in Python6:15
Create and manipulate Python lists, add items with extend and append, sort values, index from zero, slice lists, and build new lists from selected elements.
Create Tuples and Dictionaries in Python2:41
Create loops and functions in Python6:56
Learn to use for and while loops in Python, iterate lists and ranges, apply if conditions, print results, and define and call functions that return values or append to lists.
Introduction to Pandas Library in Python0:51
Import and use Pandas in Python7:43
Introduction to NumPy0:17
Import and use NumPy in Python2:57
Configure PySpark development environment to run Spark4:07
Course Rating0:16

Create Resilient Distributed Dataset (RDD) in Spark4:15
Import data files in PySpark4:15
Show data in datasets3:35
Explore your dataset in PySpark by viewing top rows with head, retrieving the first and last records with first and tail, and inspecting data types with dtypes. Then summarize data with describe and list columns with df.columns, setting up data cleaning in the next tutorial.
Clean and transform data part_1 (replace nulls with values)5:25
Clean and transform a PySpark data frame by replacing nulls with unknown, creating a clean country column, and dropping the original column; set proper data types and remove dollar signs.
Clean and transform data part_2 (remove null values)2:04
Use the filter function to remove rows with null item type, compare data before and after, and plan to replace nulls or convert the string column to float.
Clean and transform data part_3 (change data types)6:24
Clean and transform data part_4 (replace numeric nulls by the column mean)6:37
Clean and transform data part_5 (change columns names)1:50
Clean and transform data part_6 (change columns order in a dataset)0:53
Course Rating0:16

A reminder to upload data and run configuration code1:49
Upload your CSV data file to the notebook before analysis, and run all to initialize the session; re-upload and run all after any disconnection to avoid errors.
Data analysis part_1 (find sales volume per item using sum function)3:20
Data analysis part_2 (find number of orders per item using count function)2:57
Data analysis part_3 (find itemType data using aggregation functions)2:03
Data analysis part_4 (add a calculated column to datasets)2:47
Data analysis part_5 (find total profit per item_type and Sales_channel)2:44
Data analysis part_6 (find sales facts per country using conditions)4:11
Data analysis part_7 (find total profit per country)3:09
Data analysis part_8 (find countries generated total profit greater than 4 M)1:14
Run data analysis with SQL Part_12:43
Run data analysis with SQL Part_22:30
Course Rating0:16

Requirements

Simple background in data analysis

Description

Apache Spark is one of the most powerful tools used in big data analysis because:

It’s Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.

· It can run real and semi-real time data analysis.

· It can handle large scale of data.

· It can be run using simple code in Python programming language.

You can use the easy commands in Python and SQL languages, to run data analysis on big data that cannot or difficult to import inside relational database engines. This combination of Spark, Python and SQL create a powerful work environment to analyze big data easier and faster.

In this course, you will learn: What is Spark, how does it run, and how data are stored in Spark work environment. You will learn how to configure Python programming environment to run Spark code. Also, you will learn performing data analysis using real big data. In addition, you will learn to import big data files inside Python. You will learn to clean and transform data for analysis purpose. You will learn conducting business analysis using several Spark functions. You will learn to create SQL queries inside PySpark to run data analysis. After that you will learn how to interpret the results from business perspective.

Who this course is for:

Those who have need to learn data analysis in PySpark
Those who need to use SQL on Big Data

Learn Big Data Analysis with PySpark

What you'll learn

Explore related topics

Course content

Introduction5 lectures • 15min

Introduction to Python Development Environment11 lectures • 36min

Cleaning and Transforming Data in PySpark10 lectures • 36min

Performing Data Analysis in PySpark12 lectures • 30min

Requirements

Description

Who this course is for: