Learn Big Data Analysis with PySpark

Explore Google Colab as a zero-configuration online platform for Python Spark, offering free GPUs, easy sharing, and quick coding, with notes on saving work to avoid data loss when disconnected.

Log in to Colaboratory with a Gmail account, open a new Python notebook, and learn to navigate the easy Colab interface to type code in cells.

Create and manipulate Python lists, add items with extend and append, sort values, index from zero, slice lists, and build new lists from selected elements.

Explore creating and using tuples and dictionaries in Python, including tuple immutability, indexing to extract values, and dictionary key-value syntax with retrieval by key.

Learn to use for and while loops in Python, iterate lists and ranges, apply if conditions, print results, and define and call functions that return values or append to lists.

Explore Pandas, a Python library for data manipulation and analysis, focusing on data structures, operations for numerical tables and time series, and how to import it.

Import pandas and load a CSV into a data frame to inspect data with head, tail, shape, and columns. Count values and plot a bar chart using pandas plotting.

Explore NumPy, the Python library for working with arrays and multi-dimensional arrays like matrices, and learn its functions for array operations.

Import NumPy in a notebook and alias it, create a normal distribution dataset (mean 5, std 10, 50 values), compute its mean, and plot a histogram with Matplotlib.

Configure a Google Colab environment to run Spark with PySpark by installing the Java Development Kit and Spark 3.2.1, setting Java Home and Spark Home, and initializing a Spark session.

Create a resilient distributed dataset (RDD) in Spark by parallelizing Python data, then build and display Spark data frames using createDataFrame and df.show.

Learn how to import a csv file in PySpark, create a data frame in a notebook or CoLab, and inspect the dataset using spark.read.csv with header true and df.show.

Explore your dataset in PySpark by viewing top rows with head, retrieving the first and last records with first and tail, and inspecting data types with dtypes. Then summarize data with describe and list columns with df.columns, setting up data cleaning in the next tutorial.

Clean and transform a PySpark data frame by replacing nulls with unknown, creating a clean country column, and dropping the original column; set proper data types and remove dollar signs.

Use the filter function to remove rows with null item type, compare data before and after, and plan to replace nulls or convert the string column to float.

Learn to clean and transform data with PySpark by creating float columns for unit_salt, unit_price, and unit_cost, using withColumn, cast, and substring, then drop originals and replace nulls with mean.

remove duplicates to clean the data frame, overwrite the original data, rename 'Clean Country' to 'country', and reorder columns to start with order_id, region, country, sales channel, and order date.

Learn to reorder columns with select and overwrite the dataframe by listing: order ID, region, country, item type, sales channel, order date, ship date, unit sold, unit price, unit cost.

Upload your CSV data file to the notebook before analysis, and run all to initialize the session; re-upload and run all after any disconnection to avoid errors.

Learn to compute sales volume per item in PySpark by aggregating units sold with sum, grouping by item type, and sorting results in descending order.

Compute the number of orders per item type by grouping by item type and counting order IDs, then sort by descending orders to reveal top-demand items.

Group by item type in a spark dataframe and apply sum, average, and max aggregations to compute total units sold, average unit price, and maximum item cost.

Learn to add calculated columns to a dataframe, computing total sales and total profit from unit sold, unit price, and unit cost for item types per sales channel region analysis.

Compute total profit by item type and by sales channel using group by and sum, then sort descending to reveal cosmetics and offline channel as top profit drivers.

Filter sales data by country using PySpark data frames, exploring exact matches with equals and wildcards, and inspecting Japan, India, and Indonesia to compare orders, sales, and profits.

Learn how to filter a data frame by country and item type, including not-equal conditions, to compute total profit per country using group by and sum in PySpark.

Learn to create a data frame of total profit per country and filter for countries with profits greater than 4 million using PySpark.

Register a temporary table in Spark, then run Spark SQL queries to analyze data with PySpark, selecting columns and filtering by region such as Europe.

Run sql queries in spark to sum total cells by region and by country, group and order results descending, highlighting Germany, Italy, Canada, and France.