Master Google Cloud Dataflow and Apache Beam

Explore big data concepts, from volumes and structured or unstructured data to parallel processing, validation, cleansing, transformation, and aggregation, with emphasis on chunking data and merging results in Apache Beam.

Explore Apache Beam and Google Cloud Dataflow, the open source model for batch and streaming data, with direct and Spark runners, multi-language support, and built-in templates, SQL, and ML capabilities.

Launch your first Apache Beam program by setting up a Maven project, reading invoice, item, and store data with text IO, and mapping records on the direct runner.

Explore how CoGroupByKey and group by key in Apache Beam join invoices and items by person id and item id to enrich records with item names.

Explore how combine enables gradual aggregation and contrast with group by key, then implement a custom CombineFn to compute per-key averages by maintaining a serializable accumulator with sum and count.

Learn how to use side inputs in Apache Beam to enrich a main dataset with auxiliary data, such as adding store names to items via a PCollection view.

Explore schema transforms in Apache Beam, including select, group, join, and filter, using an invoice dataset to demonstrate field selection, flattening, aggregation, and sql-like predicates.

Learn how Apache Beam metrics work through a Java example: counters, distribution, and gauge counters track totals, skipped items, and amount distributions, with metrics results shown after the pipeline runs.

Apply fixed time windows to group keyed data within each 60-second interval, attach timestamps to records, and write the grouped results to a CSV file.