Learn By Example : Apache Flink

Explore the requirements of streaming architecture, including buffering with no data loss, low latency, decoupled sources via a message transport queue, data consistency, and exactly-once processing.

Install Flink by downloading the binary, extracting it, and configuring environment variables; then run a local cluster with the job manager, task managers, and the dashboard, testing with examples.

Set up a Java Maven project and add Flink dependencies—data stream API, dataset API, and Flink runtime—to pom.xml, then configure for local execution instead of a cluster.

Learn Flink's data representations: unbounded streams and bounded data sets as immutable collections of entities, transformed by map and other operations, then grouped into keyed streams.

Write your first flink program by setting up the execution environment, reading a data stream from text or host and port sources, applying transformations, and writing output to sinks.

Learn how to use the map and flatMap operations in Apache Flink: extract a list of specializations, then emit each specialization as a separate output using flatMap.

Differentiate stateless transformations that operate on a single stream item from stateful transformations that accumulate across multiple stream entities, using windows to define subsets and enable sums and other aggregates.

Apply a keyBy transformation to group the stream by specialization, then perform per-key stateful operations to count occurrences and emit a result per key.

Apply number aggregations on a data stream by key using a sum and other numeric operations, and use map transformations on tuples to prepare outputs for the sink.

Demonstrate the reduce operation in Apache Flink by mapping courses to specialization, length, and count; key by specialization; use reduce to sum and then compute the average length per specialization.

Explore stateful transformations in apache flink by applying windowed operations to keyed streams, defining subsets of data with sliding, session, count, and global windows.

Compare keyed and non-keyed windows and illustrate sliding and tumbling windows with fixed durations and time-based grouping.

Explore how count windows define window size by a fixed number of entities in a keyed stream, with each key creating its own windows and stateful operations.

Trigger new session windows when the gap between events exceeds a specified value, enabling idle-time processing on a socket stream with a 10-second gap.

Explore how the window API in Apache Flink uses window assignment, triggers, and evaluation functions to decide when windows start and end, with customizable components.

Explore implementing a custom window function in Apache Flink by defining an evaluation function, applying a count window, and producing a US signup map from the last five events.

Learn how Apache Flink handles event time, ingestion time, and processing time, and how the chosen notion governs windowing, event order, and the use of timestamp extractors and watermarks.

Connect a twitter streaming data source, parse each input to extract locations, and count occurrences within a window of 5 using a window transformation.

Build a custom Flink data source that fetches stock prices from the Yahoo Finance API for symbol, emitting events with symbol, price, and previous closing price via the source context.

Explore how Apache Flink uses value state to build stateful transformations on a keyed stream, counting words and printing results when the word print appears.

Explore how to use a list state object to accumulate distinct words seen in a streaming input and display the evolving list as new words arrive using a flatMap transformation.

Accumulate values in a reducing state in a Flink stream, initialize and update the state with a rich map function, and output the running count to the screen.

Explore how to enable checkpointing in Apache Flink, configure intervals and backends, and design restart strategies to achieve exactly-once or at-least-once processing with state recovery.

Learn how to use the union operation to combine two streams into a single stream, producing a sum stream and a product stream from input numbers, and emitting labeled results.

Join two streams in Apache Flink using a key to produce combined outputs within a 30-second tumbling window. Create a sum stream and a product stream and define key selectors.

Combine two streams by key with a coGroup operation in Apache Flink, using a 30-second window to aggregate hours and production data and compute each employee's average productivity.

Learn how to use the split operation in Apache Flink to divide a single input stream into labeled output streams using an output selector and labels, enabling conditional routing.

Explore how Apache Flink applies transformations to both data streams and data sets, using map, keyBy (groupBy), and aggregation to compute sums, counts, and ratios across entities.