Kubernetes for Data Engineering: Hands on End to End Guide

Name: Kubernetes for Data Engineering: Hands on End to End Guide
Rating: 3.8 (14 reviews)

Boost the efficiency of your Data Engineering Solutions by Deploying them on Kubernetes Cluster.

Created byGaniyu Yusuf

Last updated 4/2024

English

What you'll learn

Understand the core concepts of Kubernetes, including pods, services, deployments, and more. Learn how to set up and manage a Kubernetes cluster
Gain practical experience in deploying and managing the Kubernetes Dashboard, a powerful tool for managing Kubernetes clusters through a user-friendly interface
Learn how to deploy Apache Airflow in a Kubernetes environment. Understand how to schedule and monitor data pipelines efficiently using Airflow.
Dive into the world of Directed Acyclic Graphs (DAGs) and learn how to create, schedule, and monitor them in an Airflow environment running on Kubernetes.
Understand how to secure your Kubernetes cluster and monitor its performance. Learn about Kubernetes namespaces, RBAC, secrets, and network policies.
Learn how to scale your data applications and ensure high availability within your Kubernetes cluster.
Develop skills to troubleshoot common issues in Kubernetes and optimize the performance of your data pipelines.

Course content

6 sections • 29 lectures • 2h 18m total length

Introduction1:59
Explore how Kubernetes powers modern data engineering, manage clusters with the dashboard and terminal, and deploy Airflow and Spark to run, connect, and scale data jobs.
What this course covers2:03
Kubernetes Architecture Explained5:38
Explore Kubernetes architecture with masters and worker nodes, API server, scheduler, controller manager, and how pods, deployments, services, volumes, config maps, and secrets are managed by kubelet and container runtimes.
KubeProxy and Container Runtime Deep Dive1:00
Explore how kube-proxy manages node-level network rules to forward and balance cluster services, with modes like user space, iptables, and ipvs, and examine container runtimes that execute containers.
Kubernetes Additional Services0:51
Explore Kubernetes additional services like kube DNS and kube dashboard ui, and review cluster information such as service and namespace, plus metric collector ipstar.
Kubernetes Networking Fundamentals1:39
Discover Kubernetes networking fundamentals, including how network address translation enables port communication. Examine how containers, ports, services, deployments, config maps, secrets, and persistent volumes interconnect, with NAT shaping in-flight headers.
Kubernetes Core Concepts1:17
Explore Kubernetes core concepts like clusters, masters, nodes, and namespaces, and see how the control plane coordinates resources such as cpu and ram across the cluster.
Kubernetes Behind The Scenes0:59
Discover how kubectl and the api server validate manifests, publish objects through the api request loop into etcd, and how the controller manager deploys replica sets and pods.
Getting Started with Tools for Kubernetes3:32
Explore Kubernetes tools for data engineering from a Collabnet curated list, including cheat sheets, labs, a periodic-table overview, and AI workloads on Kubernetes with Kubeflow.
How Kubernetes can help you as an engineer4:29
Explore ten practical ways Kubernetes enhances data engineering, from scalable, self-healing pipelines to secure, portable, stateful deployments, observability, and end-to-end ci/cd orchestration.

Installing Docker Desktop2:10
Install Docker Desktop on your machine, selecting Mac with Apple Silicon or Intel and downloading the correct package. Explore Docker Desktop to manage containers and images and enable Kubernetes.
Setting up, enabling and verifying Kubernetes2:31
Learn to enable and verify Kubernetes on docker desktop by configuring resources, applying changes, and restarting. Validate the kubernetes dashboard and select the correct docker desktop context for kubectl.
Installing Cluster Managers on all Operating Systems5:03
Install and validate kubectl and other cluster management tools across Windows, Linux, and macOS, using package managers like Chocolatey, Winget, and Homebrew, with architecture-specific binaries and optional troubleshooting.
Cluster Manager Commands4:08
Master kubectl cluster management by verifying versions, cluster info, and core dns. Switch contexts and set up the Kubernetes dashboard to visualize deployments, services, and secrets.
Installing and Setting up Helm Charts2:15
Learn to install and configure helm charts for Kubernetes, including installing kubectl, using brew or alternative package managers, and building from source when needed.

Deploying Kubernetes Dashboard with Helm Charts3:40
Deploy the Kubernetes dashboard with Helm charts and a YAML configuration, configure namespace and RBAC, and access the UI via port forwarding and token-based authentication.
Generating Tokens for Kubernetes Dashboard8:16
Generate a Kubernetes dashboard token by creating an admin service account, cluster role binding, and secret, then apply resources and sign in with kubectl.
Working with Kubernetes Dashboard - End to End6:37
Navigate the Kubernetes dashboard to manage namespaces, workloads, services, ingresses, config maps, secrets, and storage, while configuring RBAC and deploying Apache Airflow on Kubernetes for end-to-end data engineering.

Deploying Apache Airflow on Kubernetes6:27
Deploy Apache Airflow on Kubernetes using Helm, create a namespace, monitor jobs like Airflow migrations and create a user, adjust resources, and troubleshoot port forwarding and web server readiness.
Upgrading and Applying Changes to Apache Airflow using Helm Charts5:30
Upgrade and apply changes to Apache Airflow using Helm charts, override sensitive keys and the executor to Kubernetes, manage values.yaml, and verify the UI connectivity.
Creating and Deploying Airflow DAGS to Kubernetes Cluster13:10
Deploying and Working with Multiple DAGS on Kubernetes Cluster17:45
Deploy and manage multiple dags on a Kubernetes cluster to fetch a CSV, attach headers, convert to JSON, push to XCom, and preview category sales with pandas.
Optimising your Airflow DAG pipeline on Kubernetes5:04
Optimize your Airflow DAG pipeline on Kubernetes by managing imports, restarting the Apache Airflow cluster, and validating data through logs and previews.

Preparing Spark Jobs to run on Kubernetes6:19
Prepare and test a simple Spark job for Kubernetes by setting up Spark and PySpark in a virtual environment, building a hello world app, and verifying locally.
Packaging your Spark Jobs on Kubernetes8:16
Package your PySpark script into a Docker image for a Kubernetes spark cluster using a Dockerfile and Bitnami Spark. Build and push to Docker Hub, then deploy to the cluster.
Deploying Spark Jobs to Kubernetes Cluster14:42
Learn how to submit Spark jobs to a Kubernetes cluster using spark-submit, configuring master, deploy mode, executor settings, and docker image, and monitor progress in the Kubernetes dashboard.
Fixing Potential Bugs during Spark Deployment on Kubernetes Cluster2:47
Fix common spark jar path errors on Kubernetes by setting the hive directory path in the configuration, resubmitting the job, and monitoring progress in the UI.

Requirements

Basic Programming Experience

Description

This is a Kubernetes For Data Engineering practical hands-on course based on a lot of requests by students.

Are you ready to elevate your data engineering skills to the next level?

This course has been meticulously designed to help you immerse yourself into the world of Kubernetes, the powerful tool revolutionizing the management of containerized applications. Join us in this comprehensive course where we explore Kubernetes and its practical applications in the realm of data engineering.

This course is suitable for all levels of experience from beginners to expert as it has been designed to equip you with essential knowledge and hands-on experience.

Here are what you'll learn:

Understanding Kubernetes: Explore the fundamentals of Kubernetes, including its architecture, core concepts, and additional services, to grasp its significance in modern data engineering.
Kubernetes Deployment: Learn how to set up Kubernetes on Docker, master kubectl for cluster management, and deploy the Kubernetes Dashboard for efficient cluster administration.
Exploring Kubernetes Components: Dive into Kubernetes components such as Kubelet, KubeProxy, container runtimes, and additional services to gain a comprehensive understanding of their roles in the Kubernetes ecosystem.
Kubernetes Networking Fundamentals: Delve into the networking fundamentals of Kubernetes to understand how containerized applications communicate within a Kubernetes cluster.
Harnessing Kubernetes for Data Engineering: Discover how Kubernetes can empower you as a data engineer, streamlining processes, enhancing scalability, and facilitating efficient management of data workflows.
Setting Up Kubernetes on Docker: Start from the basics as we guide you through setting up Kubernetes on Docker. Perfect for newcomers or those looking to refresh their understanding.
Mastering kubectl: Learn the ins and outs of kubectl, the command-line tool for managing Kubernetes clusters. Gain proficiency with essential commands and expert tips for seamless navigation.
Deploying the Kubernetes Dashboard: Follow step-by-step instructions to deploy the Kubernetes Dashboard, an intuitive interface for efficiently managing Kubernetes clusters.
Running Apache Airflow with Helm Charts: Unlock the potential of Apache Airflow, a leading tool for orchestrating complex computational workflows, by running it on Kubernetes using Helm charts.
Deploying Apache Spark on Kubernetes Cluster: Explore the deployment of Apache Spark, a powerful framework for distributed data processing, on Kubernetes. Learn how to harness the scalability and flexibility of Spark within a Kubernetes environment.

In this detailed course, you'll have easy access to each section of the course, ensuring a structured and efficient learning experience. From setting up Docker to optimizing Airflow DAGs and deploying Apache Spark on Kubernetes, we cover it all.

Join us on this journey to master Kubernetes for data engineering and take your skills to new heights.

Ready to embark on this exciting adventure? Enroll now and let's immerse ourselves into Kubernetes for data engineers together!

Who this course is for:

Everybody interested in building scalable and efficient infrastructures

Kubernetes for Data Engineering: Hands on End to End Guide

What you'll learn

Explore related topics

Course content

Introduction10 lectures • 23min

Infrastructure Setup5 lectures • 16min

Kubernetes Dashboard3 lectures • 19min

Deploying Apache Airflow to Kubernetes Cluster5 lectures • 48min

Apache Spark Deployment on Kubernetes Cluster4 lectures • 32min

Next Steps (Resources and Documentations)2 lectures • 1min

Requirements

Description

Who this course is for: