Batch Processing with Apache Beam in Python

Name: Batch Processing with Apache Beam in Python
Rating: 4.2 (63 reviews)

Easy to follow, hands-on introduction to batch data processing in Python

Created byAlexandra Abbas

Last updated 9/2020

English

English [Auto],

What you'll learn

Core concepts of the Apache Beam framework
How to design a pipeline in Apache Beam
How to install Apache Beam locally
How to build a real-world ETL pipeline in Apache Beam
How to read and write CSV data from Apache Beam
How to apply built-in and custom transformations on a dataset
How to deploy your pipeline to Cloud Dataflow on Google Cloud

Course content

3 sections • 19 lectures • 1h 9m total length

Welcome2:17
Design, build, and deploy batch data pipelines with Apache Beam in Python, leveraging extract, transform, load concepts and Beam as a unifying framework with Cloud Dataflow on Google Cloud Platform.
What is Apache Beam2:14
Beam unifies parallel data processing across APIs and translates code to Spark, Flink, or Hadoop, enabling write-once pipelines; the course uses the Python SDK for batch and streaming pipelines.
Apache Beam concepts2:06
Explore beam concepts: a pipeline defines a beam program with command-line options, while P collections enable parallel data processing and B transforms read, write, and process bounded or unbounded data.
Design a pipeline2:24
Design a pipeline that reads a CSV web analytics dataset, computes session duration from timestamps, maps IPs to countries, and averages duration by country, then writes results into fire.
Install Apache Beam2:51
Create and activate a Python 3.7 virtual environment named Apache Beam Tutorial, install Apache Beam and the Google Cloud Dataflow package to deploy pipelines to Dataflow.

Create a pipeline2:45
Create an Apache Beam pipeline inside a function and run it from a top-level if block, using a context manager to wrap transforms and set logging to warning.
Configure pipeline options3:25
Configure pipeline options in Apache Beam Python using Pipeline Options to manage custom and default arguments, specify input and output paths, and choose runners like DirectRunner or Dataflow.
Read data from CSV file3:42
Learn to read a CSV file in Apache Beam with read from text, producing a PCollection of lines, and skip the header using the pipeline.
Format data with Map6:02
Format text lines in a beam pipeline by converting each line string into a list of fields with the map transformation, splitting by commas, and print results to verify locally.
Transform data with ParDo and DoFn8:00
Use a ParDo with a DoFn to compute visit durations from start and end timestamps. Convert strings to datetimes, subtract to get seconds, and yield the IP with the duration.
Call external API from DoFn5:27
Call an external API from a DoFn in beam to map IP addresses to countries, using requests to IP API dot com and return country and IP.
Access side input7:06
Combine the IP map with duration data using a side input in Apache Beam to map each IP to its country and compute the average duration for each country.
Combine data2:23
Group data by country and compute the average session duration using Apache Beam's combine per key with the mean function in Python.
Format output2:02
Format the output by joining fields with commas, convert elements to string with a lambda map, and write the country name and average session duration to a CSP file.
Write output CSV file3:27
Finish an Apache Beam Python pipeline by writing the results to a csv file using the write to text transform, specifying an output prefix and a country-level average session duration.

What is Cloud Dataflow1:26
Explore how Google Cloud Dataflow provides a managed, serverless platform for Apache Beam pipelines, auto-scales for streaming and batch workloads, with templates and a DAG view to BigQuery.
Set up Google Cloud environment2:57
Set up Google Cloud for Apache Beam dataflow by enabling the data flow API and creating a unique regional storage bucket with staging and temp folders.
Run pipeline on Cloud Dataflow7:40
Run an Apache Beam pipeline on Google Cloud Dataflow from the command line, using cloud storage input and output, project, staging, region, and save main session.
Clean up in Google Cloud0:49
Delete the Google Cloud project used to run your dataflow job, shut it down in project settings, and ensure all resources are removed to stop charges.

Requirements

Python programming experience
Having an idea of distributed data processing e.g. You have used Spark before
Having Conda (or other Virtual Environment Manager) installed on your machine

Description

Apache Beam is an open-source programming model for defining large scale ETL, batch and streaming data processing pipelines. It is used by companies like Google, Discord and PayPal.

In this course you will learn Apache Beam in a practical manner, with every lecture comes a full coding screencast. By the end of the course you'll be able to build your own custom batch data processing pipeline in Apache Beam.

This course includes 20 concise bite-size lectures and a real-life coding project that you can add to your Github portfolio! You're expected to follow the instructor and code along with her.

You will learn:

How to install Apache Beam on your machine
Basic and advanced Apache Beam concepts
How to develop a real-world batch processing pipeline
How to define custom transformation steps
How to deploy your pipeline on Cloud Dataflow

This course is for all levels. You do not need any previous knowledge of Apache Beam or Cloud Dataflow.

Who this course is for:

Data Engineers
Aspiring Data Engineers
Python developers interested in Apache Beam

Batch Processing with Apache Beam in Python

What you'll learn

Explore related topics

Course content

Get started5 lectures • 12min

Develop a pipeline10 lectures • 44min

Deploy to Cloud Dataflow4 lectures • 13min

Requirements

Description

Who this course is for: