Learning Apache Spark | Master Spark for Big Data Processing

Learn word count with Apache Spark, turning input words into key value pairs where each word maps to its count. See how distributed computing speeds counting across machines.

Explore map and reduce concepts in big data processing with Apache Spark; learn about key-value pairs, mapper and reducer functions, and distributed word count across multiple nodes.

Explore how Hadoop combines map and reduce with distributed computing to form a fault-tolerant, scalable open-source framework for big data processing, and its core components.

Explore the apache hadoop architecture, including the name node as master, data nodes as workers, hdfs and yarn for fault-tolerant distributed storage and map-reduce processing.

Explore the differences between Apache Hadoop's disk-based, MapReduce approach and Apache Spark's in-memory, RAM-cached computation. Learn why Spark delivers faster processing and how memory vs disk impacts performance across APIs.

Explore the Apache Spark architecture and its core components - driver, cluster manager, worker nodes, SparkContext, executors, and tasks - and understand how the driver schedules work, tracks progress, and returns results.

Learn how PySpark acts as Python's API for Apache Spark, enabling distributed data processing with Python to sort, filter, and analyze data across clusters.

Install Java JDK to set up PySpark on Apache Spark, focusing on JDK 17 for Windows and configuring java_home and path with the bin folder, then verify with java -version.

Install Python and prerequisites for PySpark on Apache Spark. Download Python from python.org, customize installation, set the path, install to the C drive, and verify with python --version.

Install and use Jupyter Lab and Jupyter Notebook to connect PySpark APIs, run commands, read data, and visualize PySpark data for big data processing.

Install PySpark by running pip install pyspark to set up the Python API for Apache Spark, with Java 17 and Python 3.8+ prerequisites.

Initialize the spark session to start PySpark in a Jupyter notebook, using SparkSession.builder with a name and config, then getOrCreate to make the spark entry point available.

Set up and connect an Ubuntu 24.04 AWS EC2 instance with a key pair and security group, then configure SSH and ports to run Apache Spark via PySpark.

Explore how the Databricks community edition accelerates learning Apache spark with pre-installed libraries, notebook-based workflows, and scalable multi-language support (Python, SQL, Scala, R) that mirrors real projects.

Choose Databricks Community Edition when you want free, zero local setup, cloud-based Apache Spark access, and managed libraries with automated resource management. Weigh Databricks against local installs for library compatibility.

Learn to run and switch between Python, R, SQL, and markdown using Databricks magic commands, display HTML content, and explore the file system within a notebook.

Explore Databricks magic commands and how to list and call them with the list magic command, then use pip to install pandas and memorize command usage for Spark workflows.

Apache Spark data frames serve as distributed two-dimensional tables across the driver, cluster manager, and worker nodes, illustrated by a word count example.

Create dataframes from CSV files by reading employee and department data, and validate their schemas with print schema. Enable header=true and infer schema=true to ensure accurate data typing.

Create data frames from parquet files using PySpark, loading employee and department parquet sources with spark.read.parquet, and inspecting data with show and print schema.

Explore data transformations in Apache Spark using Spark SQL and data frames, focusing on the select command to extract employee_id, employee_name, designation, and salary.

Apply the filter data transformation to a Spark data frame by specifying a condition, such as salary greater than 70,000, to retrieve matching rows.

Explore how to use order by in Spark data frame to sort by a column, with ascending and descending options, demonstrated on an employee salary dataset.

Learn how to use groupby in Spark SQL and DataFrames to group data by a column such as department and apply aggregations like count to compute department employee totals.

Learn to use aggregate functions in Spark SQL and data frames to compute average, min, and max values from a column, importing functions from PySpark SQL.

Demonstrates inner join in spark data frames by joining employee and department data frames on the common employee_department column, producing a merged data frame that includes department_id.

Explore spark catalogs and their role in managing and querying structured data via a unified spark sql interface, hosting data as tables and views, and exposing metadata and functions.

Learn to list the catalogs in your Spark SQL session using catalog commands shown in the Pi Spark SQL documentation and understand how catalogs host databases and tables.

discover how to list databases in the spark catalog using spark.catalog.listDatabases, run spark in memory, and view the default database details.

Learn how to list tables from the current database using Spark catalog, including catalog dot list tables, default database handling, and optional search patterns.

Create a temporary view from a data frame in Spark by calling create or replace temporary view on the data frame, then verify the view in the catalog.

Run SQL queries on Spark temp views to compute averages by department and fetch max and min salaries, using group by and basic select statements.

Drop a temporary view in the Spark catalog with spark.catalog.drop and the view name, then verify the drop by listing tables to confirm no views remain.

Discover how to use Databricks utilities to programmatically manage resources such as notebooks, jobs, secrets, widgets, and more, including running notebooks with the dbutils run command.

Explore dbfs, the Databricks file system, and learn to read a CSV into a Spark data frame using Databricks file system versus local storage.

Create your own directory in the Databricks file system using dbutils mkdir, list directories, and confirm that the new directory named my directory exists.

Learn how to copy files into a Databricks file system directory using dbutils' file system copy, verify with list, and read the csv into a Spark dataframe with header true.

Explore pandas, a Python library for handling data with data frames, a two-dimensional table for storing and manipulating data, and learn how to install and start using it.

Explore how pandas runs on Apache Spark to enable distributed data processing with PySpark pandas across multiple machines via parallel execution.

learn to read and write data with pandas in a spark cluster using PySpark pandas, loading CSV and parquet files and validating with sample rows.

Learn to manipulate data with PySpark pandas by creating new columns, filtering by total amount, and sorting by order date in a sales orders data frame.

Learn to merge and join data frames in PySpark pandas by combining products and sales orders on product_id, using inner join, suffixing left and right, and distinguishing merge versus join.

apply group by on customer id using pandas on Apache Spark to compute the total amount purchased, by merging products and sales orders dataframes and aggregating sums.

Visualize sales data in PySpark pandas using the data frame plot API to create area charts with x and y axes, showing product ID insights.

Explore how Apache Spark structured streaming provides a scalable, fault-tolerant way to process streaming data with micro-batches using Spark SQL.

Spark structured streaming processes data in micro-batches by default, with 100 ms latency. Since Spark 2.3, continuous processing lowers latency below 1 ms and yields at least once fault tolerance.

Learn the programmatic approach to handle structured streaming with Apache Spark, using input and result tables updated every second and writing results to an external sink.

Discover programmatic modes in Apache Spark for structured streaming, including complete, append, and update modes to write results from input tables to the results table and external sink.

Create data frames from sales orders and products csv files in Databricks, verify they are not streaming, and introduce read stream and write stream APIs for streaming data in lectures.

Use the readStream API to ingest data into streaming data frames, write to parquet in a databricks compatible location, and read back with a defined schema.

Utilize the write stream API to join sales order and product streams by product ID, then write to a parquet table with a checkpoint location and append mode.

Query the streaming data frame with the write stream API, writing into a table. Then verify results by totaling sales and grouping by product ID to reveal product totals.

Learn to stop a Spark structured streaming query using the stop method, inspect query id, name, status, and termination details, and see how to manage streaming queries with query management.

Install and configure Apache Kafka with Apache Spark to enable structured streaming. Follow quick start steps from download to starting ZooKeeper and Kafka server, create topics, and stream events.

Learn to set up a Kafka environment with Zookeeper, create and use a topic, produce and consume messages, and read from the beginning to demonstrate structured streaming with Spark.

Integrate Apache Kafka with Spark Structured Streaming to read from Kafka sources, perform word count on streaming data, and output results to the console.

Terminate the kafka environment by stopping the producer and consumer with ctrl-c, halting the kafka broker and zookeeper, and cleaning logs and data.

Explore how Apache Spark Structured Streaming handles late data arrivals with watermarking, using event-time windows to decide whether to process or drop late events.

Explore how Apache Spark watermarking handles late data arrivals in streaming, using 10-minute windows and 5-minute aggregations to update taxi fare sums.