Apache Spark 3 - Real-time Stream Processing using Scala

Set up your Spark development environment in three steps: install JDK and Hadoop WinUtils, install Spark binaries with env vars, and configure IntelliJ IDEA to run a test Spark app.

Set up the spark development environment by installing Java JDK 8 or 11, configuring JAVA_HOME and PATH, verifying with java -version on Windows, and set up Hadoop WinUtils and HADOOP_HOME.

Install spark 3 binaries, set spark_home and path, configure pyspark_python and pythonpath, adjust log4j to warn, and verify with spark-shell and pyspark.

Set up and test your Spark development IDE by installing the Scala plugin in IntelliJ and configuring SBT, then run a hello Spark SQL example for stream processing readiness.

Install and run a local Apache Kafka instance to support Spark real-time stream processing, including setup, ZooKeeper and broker configuration, and producer-consumer testing.

Explore how Spark stream processing extends Spark dataframe APIs to handle late-arriving records, incremental computations, and job failures with restart, enabling 15-minute interval analytics.

Explore the evolution from DStream to Structured Streaming, highlighting micro-batch processing, event time support, and unified DataFrame APIs built on Spark SQL for real-time stream processing.

Create your first Spark structured streaming app by reading text from a tcp socket, counting words in real time, and printing results to the console with checkpointing.

Explore how Spark streaming runs as micro-batches, reading streams, processing with DataFrame, and writing results via a background thread, with no trigger, time-based, one-time batch, or millisecond continuous options.

Read streaming json files via spark readStream, enable schema inference, and flatten invoices by exploding line items. Write the flattened data to an output directory with checkpointing and append mode.

Explore streaming sources and sinks from a json directory, process via micro-batches with checkpointing, and apply append, update, complete output modes, including maxFilesPerTrigger.

Learn how Spark Structured Streaming achieves fault tolerance and exactly-once processing through checkpointing and write-ahead logs, enabling safe restarts with re-readable sources and deterministic outcomes.

Stream from kafka with spark structured streaming, query spark-sql-kafka, parse json from the value field, flatten invoices, and sink results to the file system.

Learn to build a Spark streaming app that reads invoices from Kafka, creates notification data, and writes to Kafka with invoice number as the key and JSON value.

Run two streaming queries in one Spark application reading invoices from Kafka once. Produce a Kafka notification and a flattened file sink output, with unique checkpoint locations and parallel execution.

Learn to read streams from Kafka, deserialize key and value as JSON, CSV, or AVRO in Spark, and write back to Kafka using to_json, to_csv, or to_avro.

Learn to write avro-encoded Kafka sink messages by converting json input with a schema, and using spark-avro to produce binary avro payloads via to_avro.

Learn to read Avro-formatted Kafka records in Spark using from_avro and an Avro schema, then compute total purchases and rewards per customer and publish notification records back to Kafka.

Compare stateless and stateful transformations in Spark structured streaming, showing how state stores enable cross-batch aggregations, memory management, and custom cleanup options.

Explore event time and windowing in real-time streams with tumbling and sliding windows. Learn how trigger time and micro-batches influence time-bound window aggregates and late arrivals.

Explains tumbling window aggregation in Spark streaming by reading Kafka data and computing buy and sell sums in 15-minute windows, including late arrivals.

Describe how watermarking handles late arriving records in Spark streaming using 15-minute windows and the withWatermark() method. Emphasize state-store cleanup and the impact on memory as windows expire.

Learn how watermark and output modes—complete, update, and append—affect time-based window aggregates, state cleanup, and sink behavior in Spark streaming.

Explore sliding window aggregates in Apache Spark 3 real-time stream processing, computing moving maximums over a 15-minute window with a 5-minute slide using event time, watermarking, and state cleanup.

Join a streaming dataframe with a static dataframe in Spark structured streaming to enrich login events with Cassandra data, and update last_login in real time via foreachBatch.

Understand stream-to-stream joins as stateful operations, using a state store to match impressions with clicks in real time. Learn about inner joins on inventory-id and the need for unique ids.

Apply watermarks to both streams to clean the state store in stream-to-stream joins. Use a 30-minute watermark on the impression and click events to bound data.

Learn to perform left and right outer joins on two streams in Spark structured streaming using a watermark and time-range constraints; unmatched records appear in the next micro-batch.

Celebrate your achievement and commit to ongoing learning as you continue exploring Spark real-time stream processing using Scala. Share feedback, requests, and reviews to guide future updates and new courses.