Apache Spark 3 - Real-time Stream Processing using Scala

Name: Apache Spark 3 - Real-time Stream Processing using Scala
Rating: 4.7 (521 reviews)

Learn to create Real-time Stream Processing applications using Apache Spark

Created byPrashant Kumar Pandey, Learning Journal

Last updated 1/2023

English

What you'll learn

Real-time Stream Processing Concepts
Spark Structured Streaming APIs and Architecture
Working with File Streams
Working With Kafka Source and Integrating Spark with Kafka
State-less and State-full Streaming Transformations
Windowing Aggregates using Spark Stream
Watermarking and State Cleanup
Streaming Joins and Aggregation
Handling Memory Problems with Streaming Joins
Creating Arbitrary Streaming Sinks

Course content

7 sections • 33 lectures • 4h 18m total length

About the Course2:58
Course Prerequisite1:19
Source Code and Other Resources0:12

Spark Development Environment1:44
Set up your Spark development environment in three steps: install JDK and Hadoop WinUtils, install Spark binaries with env vars, and configure IntelliJ IDEA to run a test Spark app.
Spark Installation Prerequisites5:25
Installing Apache Spark8:26
Install spark 3 binaries, set spark_home and path, configure pyspark_python and pythonpath, adjust log4j to warn, and verify with spark-shell and pyspark.
Setup and test your IDE6:44
Install and run Apache Kafka10:24

Introduction to Stream Processing9:16
Explore how Spark stream processing extends Spark dataframe APIs to handle late-arriving records, incremental computations, and job failures with restart, enabling 15-minute interval analytics.
Spark Streaming APIs - Dstream Vs Structured Streaming3:50
Explore the evolution from DStream to Structured Streaming, highlighting micro-batch processing, event time support, and unified DataFrame APIs built on Spark SQL for real-time stream processing.
Creating your first stream processing application17:40
Stream processing model in Spark8:34
Working with Files and Directories11:43
Streaming Sources, Sinks and Output Mode13:05
Explore streaming sources and sinks from a json directory, process via micro-batches with checkpointing, and apply append, update, complete output modes, including maxFilesPerTrigger.
Fault Tolerance and Restarts6:29

Streaming from Kafka Source16:19
Working with Kafka Sinks10:32
Learn to build a Spark streaming app that reads invoices from Kafka, creates notification data, and writes to Kafka with invoice number as the key and JSON value.
Multi-query Streams Application4:13
Kafka Serialization and Deserialization for Spark5:00
Creating Kafka AVRO Sinks4:00
Working with Kafka AVRO Source4:52
Learn to read Avro-formatted Kafka records in Spark using from_avro and an Avro schema, then compute total purchases and rewards per customer and publish notification records back to Kafka.

Joining Stream to static source13:57
Join a streaming dataframe with a static dataframe in Spark structured streaming to enrich login events with Cassandra data, and update last_login in real time via foreachBatch.
Joining Stream to another Stream9:57
Streaming Watermark7:17
Apply watermarks to both streams to clean the state store in stream-to-stream joins. Use a 30-minute watermark on the impression and click events to bound data.
Streaming Outer Joins11:17

Requirements

Spark Fundamentals and exposure to Spark Dataframe APIs
Kafka Fundamentals and working knowledge of Apache Kafka
Programming Knowledge Using Scala Programming Language
A Recent 64-bit Windows/Mac/Linux Machine with 8 GB RAM

Description

About the Course

I am creating Apache Spark 3 - Real-time Stream Processing using the Scala course to help you understand the Real-time Stream processing using Apache Spark and apply that knowledge to build real-time stream processing solutions. This course is example-driven and follows a working session like approach. We will be taking a live coding approach and explain all the needed concepts along the way.

Who should take this Course?

I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using the Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with the people who implement Apache Spark at the ground level.

Spark Version used in the Course

This Course is using the Apache Spark 3.x. I have tested all the source code and examples used in this Course on Apache Spark 3.0.0 open-source distribution.

Who this course is for:

Software Engineers and Architects who are willing to design and develop a Bigdata Engineering Projects using Apache Spark
Programmers and developers who are aspiring to grow and learn Data Engineering using Apache Spark

Apache Spark 3 - Real-time Stream Processing using Scala

What you'll learn

Explore related topics

Course content

Before you start3 lectures • 4min

Setup your Environment5 lectures • 33min

Getting started with Spark Structured Streaming7 lectures • 1hr 11min

Spark Streaming with Kafka6 lectures • 45min

Windowing and Aggregates6 lectures • 1hr 2min

Stream Processing and Joins4 lectures • 42min

Keep Learning2 lectures • 1min

Requirements

Description

Who this course is for: