Apache Flink with Scala 3

Explore the anatomy of a Flink pipeline, covering data sources, operators, sinks, and parallel processing, with focus on data streams, serialization, keyBy, and back pressure.

Study the data source component of Flink pipelines, learning APIs to read from collections, text files, sockets, and custom sources like data generators and Kafka connectors.

Write basic Flink code to read humidity data from multiple data sources, using a humidity reading case class with fromString parsing, and print results from collection and text file inputs.

Create an app to read data from a socket using a test string three, specify hostname and port, and use netcat to send data and show console output in IntelliJ.

Read data from a generator that emits subscription events, payment and cancel variants, by implementing a subscription event generator as a Flink source in Scala 3 and printing the stream.

Explore how Apache Kafka serves as a publish-subscribe data source, with topics and partitions for parallelism, and how Flink reads from Kafka using a configured source with serializers and deserializers.

Explore data transformation in Flink with Scala 3, from low-level process and rich functions to high-level data stream and relational APIs, including map, flatMap, filter, and key by.

Build a streaming word count application using Flink data stream APIs by applying flatMap to split lines, map to word-count pairs, key by words, and reduce to tally counts.

Explore writing Flink apps with low level APIs using process and rich functions to access state, timers, and runtime context, and learn when to favor map or flatMap for efficiency.

Compare the data stream api and the low level api in Apache Flink using scala 3, processing a user action stream with a lambda and a custom keyed process function.

Implement rich functions in Flink with Scala 3 by building a rich map function over a user action data stream, overriding open and close methods, and printing processing starts.

Explore combining streams in Flink with union for same-type data and Kinect for different data structures, then apply a process function to unify elements into one output.

Learn to combine multiple streams with union for same-type humidity data and connect for different types, partition by location, and map to a local summary using a cool map function.

Explore creating multiple streams from a data stream in Apache Flink with Scala 3 using a process function and side outputs to emit low humidity warnings alongside the main output.

Practice integrating Kafka with Flink by building a Kafka sink and source, running Docker containers, and sending humidity readings as strings to a topic.

Explore processing time, event time, and watermarks in Flink, and learn how timestamps, late events, and cutoffs drive timely results in streaming.

Explore how Flink assigns elements to windows to process streaming data, covering tumbling, sliding, session, and global windows, with a focus on time-based windows, window assigners, and triggers.

Engage in a hands-on Flink exercise that uses processing time tumbling windows of five seconds to group user events, apply a custom window function, and print event details.

Explore handling events with event time in Apache Flink with Scala 3, extracting event time, assigning timestamps and watermarks with a 10s tolerance, and applying a tumbling event time window.

Explore how window lifecycle governs creation, termination, and late data handling with allowed lateness and watermarks in Flink windows, including watermark coordination across partitions in a distributed stream.

Explore keyed vs non-keyed windows, implement process, reduce, and aggregate window functions, and control timing with triggers, purging triggers, and evictors for Flink windowing.

Explore tumbling windows in Flink with event time and watermarks to group deposits by currency into five-second windows, apply a custom window function, and print results.

Implement sliding windowing in Apache Flink with Scala 3 to process a deposit stream using a two-second window that slides every one second, producing overlapping results.

Implement a global window in Flink with Scala 3 to compare the latest humidity reading with the city average and emit location, humidity level, and last reading every five readings.

Explore handling multiple streams with interval joins and window joins on a common key, using time bounds and various windows (tumbling, sliding, session) to pair and transform correlated events.

Apply a window join between user action and purchase detail streams using a 10-second tumbling window, keyed by user id. Generate output strings with username, purchase amount, and timestamp.

Learn stateful streaming in Flink with key to state, storing per-user state across events, and compare hash map versus Rocksdb backends for durable, scalable processing.

Explore the five Flink state types—value, list, map, reducing, and aggregating—and learn how distributed, key-based state makes applications stateful. Access state via runtime context and state descriptors.

count deposits by currency using value state in flink with scala 3, implementing a key process function with an open method and per-currency state, then compare with map with state.

Demonstrate how Flink list state accumulates historic humidity readings per location using a process function and a list state, converting Java iterables to Scala lists and capping at ten elements.

Flink uses global, consistent checkpoints to capture local operator state and stream offsets, enabling fault-tolerant recovery. Configure frequency, storage, timeout, mode, and minimum pause between checkpoints.

Learn to implement and configure checkpoints in an Apache Flink with Scala 3 app, including enabling checkpointing every second and handling keyed state with checkpointed function and listener.

Deploy Flink on a Kubernetes cluster using application mode. Build a fat jar with sbt, containerize with docker, and deploy via a makefile workflow on microk8s.