Apache Beam | Google Data Flow (Python) | Hands on course

Learn how to read inputs with Python Apache Beam, covering text, Avro, Parquet, and in-memory collections, while exploring storage and database integrations.

Explore writing outputs to text, Avro, and Parquet in Apache Beam, detailing file path, prefix, suffix, shards, shard name template, compression, headers, schemas, and codecs.

Explore map, filter, and flatMap transformations in Apache Beam with hands-on examples, from filtering New York customers by age to counting word frequencies using flatMap and combine per key.

Apply the CoGroupByKey transformation in Apache Beam (Python) to join two PCollections by a common key, producing a single PCollection of movie names and ratings.

Learn to split a PCollection into two partitions with the partition transform by applying a partitioning function that assigns even numbers to one partition and odd numbers to the other.

Combine multiple lists into a single logical collection using the flatten transformation, created with beam.create and merged via beam.flatten to run and display results in the pipeline.

Explore how a composite P transform groups existing transforms into a single reusable unit, splitting student data by country, summing marks, and formatting results for US and India.

Learn how the ParDo transformation serves as a general purpose beam transform that takes one input and can produce zero, one, or more outputs, replacing map and flat map.

Learn how side inputs augment a part two transformation. Use multiple side inputs and a small in-memory side list to exclude customers listed in a side input file during processing.

Learn to generate multiple outputs from a ParDo transformation using side outputs and dot underscore output parameters, creating three collections: New York customers, other customers, and names starting with j.

Explore how Apache Beam processes streaming data in real time with a unified API. Learn windowing, watermarks, and triggers for late elements in streaming data from sources like Pub/Sub.

Explain the loosely coupled Google Pub/Sub publish-subscribe model with producers, topics, and subscribers; compare push and pull delivery, and outline topic and subscription setup for streaming with Apache Beam.

Demonstrates an end-to-end Apache Beam streaming workflow with Google Pub/Sub, publishing movie data, filtering comedy, and printing results via a subscriber.

Discover the windows concept in streaming with Apache Beam. See why windows are needed, use two-minute segments for aggregates, and preview tumbling, global, and session window types.

Compare event time and processing time for windowing, weighing latency against correctness; processing time may yield wrong results for late data, while event time handles lateness.

Explore tumbling and sliding time based windows in Apache Beam, learn how to define windows, perform aggregations, and understand overlapping data across intervals.

Publish ratings from ratings.csv to the ratings topic. Build an Apache Beam pipeline that creates fixed 10-second windows, counts ratings per movie, and publishes results to rating_count topic for display.

Explore implementing sliding windows in Apache Beam by converting a fixed window to a four-second sliding window with a two-second slide, counting ratings by movie ID in each window.

Learn how session windows in Apache Beam on Google Data Flow (Python) open on user activity and close after inactivity, counting movies per genre in movie dataset within 25 seconds.

Explore handling late elements in a five-minute streaming window using allowed lateness, which waits for late data up to two days and emits updated sums.

Explore how Apache Beam uses triggers to emit window results, including event time, processing time, and data driven triggers, with options for early and late results and accumulation modes.

Explore composite triggers in Apache Beam, including repeatedly (forever), after reach (in order), after first (or), after all (and), and finally, with window emission and accumulation mode.

Explore Google Cloud Dataflow, a serverless engine powered by Apache Beam. Learn how to read data, apply Beam transforms, and write to sinks, with templates, notebooks, and Beam SQL.

Learn to run a Google Cloud Dataflow template to stream Pub/Sub data into BigQuery, create a topic and table, and configure a temp bucket using the Apache Beam SDK.

Learn to use Google Cloud Dataflow notebooks to run Apache Beam code, enable APIs, create and configure a notebook instance with regional workers and pre-installed packages.

Explore Beam SQL by applying SQL on a PCollection with a defined row schema to transform data; Java SDK support exists as of 2020, with Python support coming later.

Run Google Cloud Dataflow jobs with Beam SQL to aggregate real-time taxi rides in one-minute tumbling windows, counting picked-up passengers and writing results to a BigQuery table.