Apache Beam | A Hands-On course to build Big data Pipelines

Name: Apache Beam | A Hands-On course to build Big data Pipelines
Rating: 4.5 (2094 reviews)

Build Big data pipelines with Apache Beam in any language and run it via Spark, Flink, GCP (Google Cloud Dataflow).

Created byJ Garg - Real Time Learning

Last updated 6/2025

English

What you'll learn

Learn Apache Beam - A portable programming model whose pipelines can be deployed on Spark, Flink, GCP (Google Cloud Dataflow) etc.
Understand the working of each and every component of Apache Beam with HANDS-ON examples.
Learn Apache Beam fundamentals including its Architecture, Programming model, Pcollections, Pipelines etc.
Multiple PTransforms to Read, Transform and Write the processed data.
Advance concepts of Windowing, Triggers, Watermarks, Late elements, Type Hints and many more.
Load data to Google BigQuery Tables from Apache Beam pipeline.
Build Real-Time business's Big data processing pipelines using Apache Beam.
Data-sets and Beam codes used in lectures are available in resources tab.

Course content

15 sections • 62 lectures • 5h 24m total length

Introduction to Apache Beam5:29
Introduction lecture to kick off the Apache Beam course.
Evolution of Big data Frameworks6:04
This lecture explains how over the time various Big data frameworks evolved and where does Apache Beam stands.
Architecture of Apache Beam5:53
This video explains the underlying architecture of Apache Beam.
Flow of Beam's Programming Model3:05
Flow of Apache Beam's Programming Model.
Basic Terminologies in Beam5:53
Installation8:07
Install Apache Beam using Google Colab

Structure of a Beam Pipeline5:24
This video explains the default and common structure of any Beam pipeline that we build.
Various Read Transforms in Beam8:41
Apache Beam provides us with various Read Transforms to read varied types of files like Text, Parquet, Avro etc.
Create Transform3:17
Create Transform is used to create self data in Apache Beam.
Various Write Transforms in Beam6:00
Apache Beam provides us with various Write Transforms to read varied types of files like Text, Parquet, Avro, Databases etc.
Map, FlatMap & Filter - Part 111:06
Start building first Beam pipeline using Map,FlatMap and Filter Transform.
Map, FlatMap & Filter - Part 22:25
Tweaks made in the previous pipeline.
Branching Pipelines6:23
Assignment 1
ParDo Transform8:12
The most important Transform in Beam is ParDo. It is basically the parent class of Map, FlatMap etc.
Advanced Combiner of Beam7:32
In Apache Beam, Combiner comes with more enhanced features which can do more complex computations.
Create Composite Transforms5:41
You can also create you own Transform which is a combination of multiple transforms. Composite transform of Beam is good for reusability
CoGroupBy for Joins4:09
CogroupBy is the base class for all Joins of Apache Beam.
How to access files from Google Drive0:28

Introduction to Case Study1:57
Bank wants to identify its Credit cards and Loan defaulters. We will build a Apache Beam pipeline for processing.
Requirements and Data walk-through for Card skippers4:12
Requirements and Data walk-through for identifying credit card payment skippers.
Identifying Credit card payment skippers4:27
Build Apache Beam Pipeline to identify credit card payment skippers.
Requirements and Data walk-through for Loan Defaulters2:17
Requirements and Data walk-through for identifying Loan defaulters.
Identifying Loan Defaulters - Part 19:41
Build Apache Beam Pipeline to identify Loan defaulters.
Identifying Loan Defaulters - Part 24:57
Build Apache Beam Pipeline to identify Loan defaulters - Part 2.

Introduction to Streaming2:51
What is steaming data and why is there need to process streaming data.
PubSub Streaming Architecture7:13
This lecture explains the basic architecture of Google PubSub or you can say a generalized Publisher Subscriber architecture.
Beam connection with Google Cloud0:35
Establishing Apache Beam connection with Google Cloud Platform
Setting up GCP's PubSub Project7:18
Setup Google Cloud PubSub project to run the streaming data pipeline of Beam.
Run a Demo streaming pipeline on GCP6:09
Run our first data streaming Apache Beam's Pipeline on Google Pub Sub.
Quiz 2

Introduction to Windows in Beam4:44
What are windows in any Streaming frameworks and why they are so important.
Mobile Game Example3:40
Setting up the plot to demonstrate the implementation of Windows in Apache Beam.
Time Notions in Streaming Frameworks6:14
There are various Time Notions that can be used while processing a window in Beam. This lecture explains those time notions in detail.
What are Tumbling & Sliding Windows3:20
This lecture explains the theory of Tumbling, also known as Fixed windows and Sliding Windows.
Implementing Tumbling Windows9:17
Recommendation for Windowing3:15
Implementing Sliding Windows3:03
Session Windows & its implementation5:52
Global Windows & its implementation4:16

Requirements

Basic knowledge of Distributed data processing architecture.
Basic knowledge of Python.

Description

Apache Beam is a unified and portable programming model for both Batch and Streaming data use cases.

Earlier we could run Spark, Flink & Cloud Dataflow Jobs only on their respective clusters. But now Apache Beam has come up with a portable programming model where we can build language agnostic Big data pipelines and run it on any Big data engine (Apache Spark, Flink or in Google Cloud Platform using Cloud Dataflow service and many more Big data engines).

Apache Beam is the future of building Big data processing pipelines and is going to be accepted by mass companies due to its portability. Many big companies have even started deploying Beam pipelines in their production servers.

What's included in the course ?

Complete Apache Beam concepts explained from Scratch to Real-Time implementation.
Every Apache Beam concept is taught through Hands-on, practical examples for better understanding
Core Apache Beam topics including Architecture, Various PTransforms (Map, FlatMap, Filter, ParDo etc.), Combiner, Side inputs/outputs.
ADVANCE topics - Type Hints, Encoding & Decoding, Watermarks, Triggers and many more.
Build 2 Real-time Big data case studies using Apache Beam programming model.
Learn to implement Windows functions - Tumbling, Sliding, Global and Session Windows.
Load processed data to Google Cloud BigQuery Tables from Apache Beam pipeline via Dataflow.
All codes and datasets used in lessons are attached in the course for your convenience.

Who this course is for:

Students who want to learn Apache Beam from scratch to its Live Project Implementation.
Data engineers who want to build Unified & Portable Big data processing pipelines.
Developers who want to learn a futuristic programming model for Big data processing.

Apache Beam | A Hands-On course to build Big data Pipelines

What you'll learn

Explore related topics

Course content

Introduction6 lectures • 35min

Transformations in Beam12 lectures • 1hr 9min

Side Inputs and Outputs2 lectures • 9min

Case Study - Identify Bank's Defaulter Customers6 lectures • 28min

Data encoding & decoding2 lectures • 11min

Type Hints in Beam2 lectures • 9min

Build Streaming data Pipelines5 lectures • 24min

Implement Windows in Apache Beam9 lectures • 44min

Watermarks in Streaming environment1 lecture • 8min

Triggers and its Implementation3 lectures • 17min

Requirements

Description

Who this course is for: