Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Workflow Orchestration Without Limits Using Kestra
Rating: 4.4 out of 5(2 ratings)
246 students

Workflow Orchestration Without Limits Using Kestra

Building Advanced Pipelines With Kestra on GCP
Created byAndreas Kretz
Last updated 2/2026
English
English

What you'll learn

  • Build and run production-style Kestra workflows using YAML, tasks, inputs/outputs, and orchestration logic
  • Set up Kestra locally with Docker and connect it securely to Google Cloud Platform (GCP)
  • Orchestrate scalable data pipelines on GCP using Cloud Storage (GCS) + BigQuery (load + transform with SQL)
  • Run real Python and Java code inside workflows, using Namespace Files and synchronizing flows from GitHub via Namespace and Tenant Sync
  • Design reusable workflows with subflows instead of copy-pasting logic across pipelines
  • Process dynamic inputs by looping through multiple files and controlling concurrency
  • Build faster pipelines using parallel execution for independent logic branches
  • Implement resilience with error handling, fallback logic, and retries
  • Build event-driven pipelines triggered by GCS uploads via Pub/Sub and Kestra triggers
  • Monitor and debug workflows using execution views, logs, outputs, replay, and workflow dashboards/metrics

Course content

5 sections • 15 lectures • 1h 56m total length

  • Short Introduction to the Course2:21

    A quick overview of what we’ll build, how the course is structured, and what “orchestration without limits” really means.

  • What is Kestra and Why Workflows Matter4:21

    Learn the core mental model behind Kestra flows, YAML-based orchestration, tasks, execution logic, and observability.

  • Setting Up Kestra Locally with GCP10:20

    Set up Kestra with Docker, configure Google Cloud access, and validate everything with a first end-to-end workflow.

Requirements

  • Basic familiarity with Python or SQL (you don’t need to be advanced)
  • A computer that can run Docker (Windows / macOS / Linux)
  • A Google Cloud account with access to BigQuery and Cloud Storage (free tier is enough to start)
  • Basic command line knowledge (running a few setup commands is enough)
  • Optional but helpful: Some familiarity with BigQuery or cloud concepts (projects, service accounts) & a GitHub account if you want to follow the Git sync section hands-on

Description

Orchestration is what turns “a script that works” into a pipeline you can actually run in production. In this course, you’ll learn how to build scalable, reliable workflows with Kestra on Google Cloud Platform (GCP). From local development all the way to event-driven execution in the cloud.

Kestra is a modern orchestration platform that lets you define workflows as YAML-based flows, connect tasks through inputs and outputs, and run them with built-in monitoring, retries, and execution tracking. Throughout the course, you’ll build a strong mental model for how orchestration should work, and how to design workflows that stay maintainable as they grow.

This course is fully hands-on. You’ll set up Kestra locally using Docker, connect it to Google Cloud, and start building real workflows that interact with Cloud Storage (GCS) and BigQuery. You’ll orchestrate end-to-end data pipelines: loading raw data, transforming it using SQL, and applying practical design patterns like raw vs. clean datasets and proper handling of invalid records.

You’ll also learn how to run real code inside workflows using Python and Java, how to synchronize code from GitHub using native Kestra sync tasks, and how to build reusable pipeline logic with subflows.

To push your workflows to a production-ready level, you’ll implement loops for dynamic file processing, run tasks in parallel, build resilient pipelines with error handling and retries, and create event-driven pipelines that trigger automatically when new files arrive. Finally, you’ll learn how to monitor, debug, replay executions, and build dashboards that turn workflow history into actionable metrics.

By the end, you’ll be able to build orchestration pipelines that are modular, scalable, and built for real-world operation.

Who this course is for:

  • Data professionals moving into production pipelines (from notebooks and scripts to real workflow execution)
  • Aspiring Data Engineers who want hands-on experience building real orchestration pipelines
  • Data Engineers who want to build scalable, reliable workflows on GCP with modern orchestration tooling
  • Analytics Engineers / SQL-focused engineers who want to orchestrate and automate BigQuery-based workflows
  • Anyone who wants to learn event-driven pipeline patterns, retries, monitoring, and production workflow design

Report abuse