Master Google Cloud Dataflow and Apache Beam

Explore big data concepts, from volumes and structured or unstructured data to parallel processing, validation, cleansing, transformation, and aggregation, with emphasis on chunking data and merging results in Apache Beam.

Explore Apache Beam and Google Cloud Dataflow, the open source model for batch and streaming data, with direct and Spark runners, multi-language support, and built-in templates, SQL, and ML capabilities.

Explore Apache Beam's two-part architecture—beam model written in Java, Python, or Go—and runner, and how a pipeline uses PCollection, PTransform, and IO connectors to read sources and write sinks.

Understand PCollection, an immutable, same-type data collection in Apache Beam where each element has a timestamp and can be transformed, with bounded or unbounded data via coders.

Configure Apache Beam pipelines with pipeline options to pass custom configurations, such as runner and file location, via command line arguments and main function arguments themselves.

Launch your first Apache Beam program by setting up a Maven project, reading invoice, item, and store data with text IO, and mapping records on the direct runner.

ParDo applies per-element user defined code for filtering, transforming, enriching, and computing data elements, enabling generic parallel processing in Apache Beam.

Group data by key using Apache Beam's GroupByKey to organize a single PCollection of key-value pairs for per-key aggregations with the built-in GroupByKey function.

Explore how CoGroupByKey and group by key in Apache Beam join invoices and items by person id and item id to enrich records with item names.

Explore how combine enables gradual aggregation and contrast with group by key, then implement a custom CombineFn to compute per-key averages by maintaining a serializable accumulator with sum and count.

Learn how to use multi-output to emit multiple PCollections from a single input, enabling denormalized data split into person and address outputs via a PCollection tuple in a for loop.

Partition a P collection into four sub-collections using a partition function on an invoice dataset, mapping quantities to partitions 0–3 and exporting each to files.

Learn how to use side inputs in Apache Beam to enrich a main dataset with auxiliary data, such as adding store names to items via a PCollection view.

Practice applying schema in Apache Beam by encoding and decoding POJO objects, registering Java beans, and converting CSV strings to invoice objects, demonstrating schema-driven pipeline operations and outputs.

Explore schema transforms in Apache Beam, including select, group, join, and filter, using an invoice dataset to demonstrate field selection, flattening, aggregation, and sql-like predicates.

Explore codecs in Apache Beam, learn how coders map language data types to Beam data types, and how encoding and decoding enable language-agnostic processing of byte streams.

Demonstrates using bag state with an event-time timer to attach item names to invoices, setting event-time timestamps, and grouping by event time, contrasting with processing-time timers.

Learn how Apache Beam metrics work through a Java example: counters, distribution, and gauge counters track totals, skipped items, and amount distributions, with metrics results shown after the pipeline runs.

Discover how Beam SQL lets you run SQL on bounded or unbounded data, using Calcite or BigQuery SQL with PCollection schemas and joins.

Apply fixed time windows to group keyed data within each 60-second interval, attach timestamps to records, and write the grouped results to a CSV file.

Explore sliding time windows with a 60-second duration created every 30 seconds, showing overlapping windows, data duplication, and how grouping by window writes results to sliding window.csv.

Explore session windows in dataflow and Apache Beam: a 15-second gap closes a window and new data reopens it, differentiating session windows from fixed and sliding windows.

Explore how the global window works in Apache Beam, showing that a single global window collects all data into one window and one key.