Kafka Connect - Building Data Pipelines With Kafka

This video course is about Kafka Connect and how to use it to move data between Apache Kafka and external systems.

You will start by learning what Kafka Connect is and how its core components work. I will explain the role of connectors, tasks, workers, and how Kafka Connect stores data in Kafka topics.

Since Kafka Connect needs a running Kafka cluster, you will also learn how to run Apache Kafka in Docker containers. I will show how to use basic Kafka CLI tools to create topics, list them, and consume messages.

Then we will focus on Kafka Connect.

You will learn how to:

• Download and install Kafka Connect,

• Run Kafka Connect in standalone mode using Kafka CLI and Docker,

• Run Kafka Connect in distributed mode using Kafka CLI and Docker.

I will also show how to manage Kafka Connect clusters using its REST API. You will learn how to check connector status, restart connectors, and remove them using simple HTTP requests.

We will use Kafka Connect to:

• Move data from MySQL to Kafka using a source connector,

• Move data from Kafka to PostgreSQL using a sink connector.

So, I will also show how to run MySQL and PostgreSQL databases in Docker. You’ll be able to test everything on your machine without installing anything manually.

This course is beginner-friendly. You do not need to have any prior knowledge of Kafka Connect.

By the end of this course, you will have a working data pipeline that streams data from a source database to Kafka and then from Kafka to a destination database.