Data Engineering 101 with Kestra

Kestra is an orchestration platform for engineers that enables declarative YAML workflows and API-first automation. Schedule events, run anywhere, and integrate with clouds and tools via a rich plugin ecosystem.

Explore the Kestra UI overview, from the welcome page to creating and editing flows. Navigate the flows, source and topology views, documentation, executions, dashboards, triggers, plugins, and admin settings.

Discover how flows and tasks are defined in YAML, identified by an id and namespace, and how flowable and runnable tasks orchestrate work like logs, http calls, and retries.

Explore namespaces as logical groupings for flows, like folders that organize environments, projects, teams, and departments, using dot-separated, indefinitely nestable names such as company dot engineering dot product one.

Explore Kestra architecture, comparing JDBC-based backend and Kafka with Elasticsearch for scalable, fault-tolerant orchestration, including metadata servers, schedulers, executors, workers, and enterprise deployment options.

Install kestra with docker compose by following the installation guide, verifying docker and docker compose versions, and launching the kestrel and postgres services, then access the kestrel ui at localhost:8080.

Explore diverse Kestra installations, from Docker and Docker Compose to Kubernetes clusters on AWS EKS, GCP GKE, and Azure AKS, including Podman rootless setups.

Design a flow architecture that downloads orders and products csv files from http, loads them into Postgres, performs a join, and uploads enriched orders to MongoDB, all orchestrated with Kestra.

Set up Postgres and MongoDB with Docker, running Postgres on port 15432 and MongoDB on port 27017, then create an order_details collection.

Install Kestra server by creating base64 encoded secrets for PostgreSQL and MongoDB, save them in kestra.env, update docker-compose to use the env file, and start with docker compose up -d.

Create a parallel flow with two sequential tasks to download orders.csv and products.csv via http, saving the files to internal storage and validating outputs.

Create and populate orders and products tables in a Postgres flow using http download and copy in, with csv data and a clear records step.

Create a Kestra data flow that extracts data from orders and products in Postgres, performs a join on orders.product_id = products.product_id, and uploads joined dataset to MongoDB's order details collection.

Clears MongoDB records before loading new data, deletes many records, then loads and updates data, and reviews outputs and metrics including deleted and inserted counts, and joined results.

Define plugin defaults for PostgreSQL query and copy tasks to share common attributes like URL, username, and password, reducing repetition with validated syntax and execution.

Discover triggers in Kestra, including scheduled, flow, webhook, real-time, and polling triggers. Learn their key properties, like id, type, description, disabled, and worker group key.

Explore triggers in action by building a Postgres table and a pull-based trigger that runs every 30 seconds to detect new records and trigger a flow.

Explore Kestra plugins spanning core tasks, data stores like MySQL and PostgreSQL, queues such as Kafka and Kinesis, and ELT and notification integrations, plus CDC and infrastructure plugins.

Explore how Kestra adoption across companies replaces Apache Airflow with simpler, faster, more reliable orchestration, as Leroy Merlin, Clever Cloud, and Chorus share data mesh and automated reporting success.