Master Google Cloud Dataflow and Apache Beam

Explore big data concepts, from volumes and structured or unstructured data to parallel processing, validation, cleansing, transformation, and aggregation, with emphasis on chunking data and merging results in Apache Beam.

Explore Apache Beam and Google Cloud Dataflow, the open source model for batch and streaming data, with direct and Spark runners, multi-language support, and built-in templates, SQL, and ML capabilities.

Explore Apache Beam's two-part architecture—beam model written in Java, Python, or Go—and runner, and how a pipeline uses PCollection, PTransform, and IO connectors to read sources and write sinks.

Understand PCollection, an immutable, same-type data collection in Apache Beam where each element has a timestamp and can be transformed, with bounded or unbounded data via coders.

Configure Apache Beam pipelines with pipeline options to pass custom configurations, such as runner and file location, via command line arguments and main function arguments themselves.

Launch your first Apache Beam program by setting up a Maven project, reading invoice, item, and store data with text IO, and mapping records on the direct runner.

Learn how to create custom pipeline options in Apache Beam, define getters and setters, and pass arguments to override default values in a pipeline.

ParDo applies per-element user defined code for filtering, transforming, enriching, and computing data elements, enabling generic parallel processing in Apache Beam.

Explore ParDo in depth within a Cloud Dataflow and Apache Beam pipeline, reading input, converting to map, calculating durations, and studying the DoFn life cycle from setup to tear down.

Group data by key using Apache Beam's GroupByKey to organize a single PCollection of key-value pairs for per-key aggregations with the built-in GroupByKey function.

Master cogroupbykey in apache beam with a practical workflow that groups invoices by person and item, assigns keys, and attaches item names to each invoice.

Explore how CoGroupByKey and group by key in Apache Beam join invoices and items by person id and item id to enrich records with item names.

Learn how the flatten transform merges multiple PCollections into a single logical PCollection, enabling efficient processing of data from diverse sources and iterable elements in Java.

Discover how Apache Beam's combine transform aggregates collection elements, using prebuilt and custom functions with accumulators, and how incremental combining reduces data movement compared to group by.

Explore how combine enables gradual aggregation and contrast with group by key, then implement a custom CombineFn to compute per-key averages by maintaining a serializable accumulator with sum and count.

Learn how to use multi-output to emit multiple PCollections from a single input, enabling denormalized data split into person and address outputs via a PCollection tuple in a for loop.

Partition splits a single PCollection into multiple PCollections based on a partitioning function, with partitions determined at graph construction time, and contrasts with flatten by combining or splitting collections.

Partition a P collection into four sub-collections using a partition function on an invoice dataset, mapping quantities to partitions 0–3 and exporting each to files.

Learn how to use side inputs in Apache Beam to enrich a main dataset with auxiliary data, such as adding store names to items via a PCollection view.

Use side input to enrich invoices by loading a store map as a view from the store PCollection and adding the store name to each invoice via the process context.

Practice applying schema in Apache Beam by encoding and decoding POJO objects, registering Java beans, and converting CSV strings to invoice objects, demonstrating schema-driven pipeline operations and outputs.

Explore schema transforms in Apache Beam, including select, group, join, and filter, using an invoice dataset to demonstrate field selection, flattening, aggregation, and sql-like predicates.

Explore how Apache Beam schema select extracts invoice number and item ID from a complex invoice, then flattens the schema for a single-row CSV output.

Learn to group data in a schema using the group function, by person id and item id, and sum quantity and amount into a group.csv output.

Learn how to join datasets with schema in Google Cloud Dataflow and Apache Beam, assign timestamps and keys, keep latest per key, and perform inner joins with item details.

Explore codecs in Apache Beam, learn how coders map language data types to Beam data types, and how encoding and decoding enable language-agnostic processing of byte streams.

Explore how value state provides per key storage in a dataflow pipeline, enabling read, initialization to zero, and per key counting across multiple instances and machines.

Demonstrates using bag state with an event-time timer to attach item names to invoices, setting event-time timestamps, and grouping by event time, contrasting with processing-time timers.

See how bag state and back state coordinate with a processing-time timer to attach item names to invoices, using item ids and fewer file reads.

Learn how Apache Beam metrics track the pipeline state in real time, including counters, distributions, gauges, and how names, namespaces, step-level scoping, and graceful degradation affect reporting across runners.

Learn how Apache Beam metrics work through a Java example: counters, distribution, and gauge counters track totals, skipped items, and amount distributions, with metrics results shown after the pipeline runs.

Discover how Beam SQL lets you run SQL on bounded or unbounded data, using Calcite or BigQuery SQL with PCollection schemas and joins.

Learn to integrate SQL into your Apache Beam pipelines with Beam SQL, reading invoices and items, converting to schema objects, and joining them to produce flattened results.

Apply fixed time windows to group keyed data within each 60-second interval, attach timestamps to records, and write the grouped results to a CSV file.

Explore sliding time windows with a 60-second duration created every 30 seconds, showing overlapping windows, data duplication, and how grouping by window writes results to sliding window.csv.

Explore session windows in dataflow and Apache Beam: a 15-second gap closes a window and new data reopens it, differentiating session windows from fixed and sliding windows.

Explore how the global window works in Apache Beam, showing that a single global window collects all data into one window and one key.

Learn how watermark and triggers govern windowed data in Apache Beam, including event time, processing time, and data-driven triggers for unbounded streams, with accumulation and discarding modes.