
Introduce the future of big data with Apache Beam and explain why it will be everywhere, as Navdeep Kaur shares his background and enthusiasm.
Explore Apache Beam, a unified, portable model to write a single big data pipeline for batch and streaming data and migrate it across engines like Spark or Dataflow.
Explore big data concepts through hands-on learning of Apache Spark and the Hadoop ecosystem, including Sqoop, Hive, Flume, and Cassandra.
Combine multiple lists into a single logical collection using the flatten transformation, created with beam.create and merged via beam.flatten to run and display results in the pipeline.
Learn how side inputs augment a part two transformation. Use multiple side inputs and a small in-memory side list to exclude customers listed in a side input file during processing.
Learn to generate multiple outputs from a ParDo transformation using side outputs and dot underscore output parameters, creating three collections: New York customers, other customers, and names starting with j.
Explain the loosely coupled Google Pub/Sub publish-subscribe model with producers, topics, and subscribers; compare push and pull delivery, and outline topic and subscription setup for streaming with Apache Beam.
Publish ratings from ratings.csv to the ratings topic. Build an Apache Beam pipeline that creates fixed 10-second windows, counts ratings per movie, and publishes results to rating_count topic for display.
Learn how session windows in Apache Beam on Google Data Flow (Python) open on user activity and close after inactivity, counting movies per genre in movie dataset within 25 seconds.
Apache Beam is future of Big Data technology and is used to build big data pipelines. This course is designed for beginners who want to learn how to use Apache Beam using python language . It also covers google cloud dataflow which is hottest way to build big data pipelines nowadays using Google cloud.
This course consist of various hands on to get you comfortable with various topics in Apache Beam.This course will introduce various topics:
Architecture
Transformations
Side Inputs/Outputs
Streaming with Google PubSub
Windows in Streaming
Handling Late elements
Using Triggers
Google Cloud Dataflow
Beam SQL / Beam SQL on GCP
By the end of this course, you will find yourself ready to start using Apache Beam in real work environment.
What make this course unique - it's concise that's in only 3 hours you will be able to complete it, covers all relevant topics and slides and presentations are really very exciting and easy to understand.
Why Apache beam is future of Big Data?
1. It runs on top of popular big data engine like spark, flink, Google data flow.
2. It is used by big giant like Google.
3. It solves the industry biggest problem of migration and unification from one processing engine to another.
So if you want to learn future technology , then you are right place.