Data Engineering with Google Dataflow and Apache Beam on GCP

Set up Apache Beam in Google Colab by installing the interactive package and importing the Beam library. Create a pipeline, chain steps with pipe, and run the pipeline synchronously.

Learn how to use the write transform to output data to text in Beam, with shard controls and options for Pub/Sub and BigQuery sinks, plus file path and formatting parameters.

Explore the beam map and flatMap transforms in dataflow: map processes every input record, splitting csv lines and converting fields to integers; flatMap flattens unstructured text into tokens for counting.

Demonstrate using the beam filter with a lambda predicate to keep CSV records, filtering flights from LAX and filtering a list by membership.

Apply the beam flatten function to union multiple string collections in a single pipeline, creating a combined collection and printing the unified data.

Use beam.CombinePerKey to aggregate flight delays by destination airport, summing delays per key after filtering positive delays, and forming key value pairs from flight data.

Learn how to use beam.combiners.Count.PerKey to count delays per destination airport by creating key-value pairs and applying the per key count, replacing the previous sum.

Learn how beam.CoGroupByKey joins two PCollections by a single key, building a dictionary of delay time and delayed number, then producing a joined delay table.

Learn to create custom ParDo functions in Python by wrapping logic in a DoFn class with a process method for parallel processing in Beam, returning a single record.

Set up the Google Cloud environment by creating a new project and activating Dataflow APIs, then run Apache Beam pipelines locally and in Dataflow, outputting to Cloud Storage and BigQuery.

Create a service account and generate a json key for server-to-server authentication, then create a data flow course bucket in a chosen region with standard storage and folders.

Set up the local Apache Beam Python SDK to build and test templates for Google Dataflow by installing apache-beam and apache-beam[gcp] and ensuring Python 3.7+.

Explore how to run an Apache Beam pipeline with the direct runner locally and save results to Google Cloud Storage, using a service account and GCS path configuration.

Dataflow batch job template using Apache Beam with pipeline options from a dictionary, configure project, runner, and storage locations, deploy to Cloud Storage, and test the template.

Deploy the Apache Beam pipeline to the Dataflow runner by creating a batch job from a custom template, configure the temporary location, and monitor the DAG, status, and auto-scaling.

Create a Dataflow template to write processed records to BigQuery, converting to simple key-value dicts and defining the table schema, path, and disposition options, then deploy and run.

Deploy a Dataflow pipeline to ingest from Google Cloud Storage, transform, and write to BigQuery using a template; clone and run batch jobs, monitor progress, and query the resulting table.