Kafka Connect - Building Data Pipelines With Kafka

Download Apache Kafka from the site, unzip and rename the folder, and move it to your home directory to set up Kafka home for later starting Kafka and Kafka Connect.

Learn how to use Kafka Connect to read from MySQL and write to other sources, by running MySQL in a Docker container, creating tables, inserting data, and executing SQL queries.

Create a MySQL database in Docker, set up user_profiles with auto-increment id and last_updated timestamp, and insert and update records to demonstrate Kafka Connect tracking updates.

Run postgresql in a docker container to support kafka connect data pipelines, create a database, insert sample records, and run sql queries while exposing port 5432.

Connect to a PostgreSQL database in a docker container, create a users database and a user_profiles table with an auto-incrementing id and last_updated timestamp, then insert a record.

Start a three-broker Apache Kafka cluster in Docker containers using a Docker Compose file, with Confluent images, port mappings 9092, 9094, 9096, and local volumes for data.

Start a three-broker Kafka cluster in Docker on Mac using a compose file and environment file, then create and describe a topic with the Kafka topics script.

Start a three-broker Apache Kafka cluster on Windows with Docker Compose, and verify connectivity by creating and describing a topic via brokers on ports 9092 and 9094.

Discover how Kafka Connect uses source and sink connectors to read from source databases and write to destination sources, via Confluent Hub and Debezium.

download the JDBC connector from Confluent Hub and install it as a self-managed plug-in for Kafka Connect, then add the MySQL or PostgreSQL JDBC driver to the lib folder.

Download the MySQL JDBC connector from Maven repository and move the jar into the Kafka Connect JDBC connector's lib folder, then configure Kafka Connect to use MySQL for data pipelines.

Learn to integrate PostgreSQL with Kafka Connect by downloading the PostgreSQL JDBC driver, updating the JDBC connector via Maven repository, and configuring Kafka Connect to work with PostgreSQL alongside MySQL.

Explain standalone mode, a single-process setup on one machine without fault tolerance, and distributed mode, offering scalability and automatic failover. Configure with worker and connector properties for development use.

Configure a Kafka Connect worker in standalone mode by editing connect-standalone.properties, focusing on bootstrap.servers and plugin.path to load connectors from the specified folder.

Prepare a MySQL source connector configuration for Kafka Connect in standalone mode, including name, connector class, tasks max, URL, mode, timestamp and incrementing columns, topic prefix, and table whitelist.

Run Kafka Connect in standalone mode after starting Kafka cluster and MySQL server. Configure with connect standalone properties file and MySQL JDBC properties file to stream changes to Kafka topics.

Test the Kafka Connect source connector in standalone mode by connecting to a MySQL database in Docker and producing and consuming insert and update events to a Kafka topic.

Explore how to use a sink connector to read data from a Kafka topic and store it in a destination database, including MySQL, Oracle, Microsoft SQL Server, or PostgreSQL.

Configure a Kafka Connect sink for PostgreSQL by creating a JDBC sink connector and standalone properties, mapping Kafka topics to a user profiles table with upsert semantics.

Run kafka connect in standalone mode with source and sink connectors, wiring mysql to postgresql via a kafka topic, using command line interface scripts and docker for setup.

Learn to run kafka connect in docker compose standalone mode with a three-broker kafka cluster, exposing rest api on port 8083 and managing connectors and plugins via mounted volumes.

Configure Kafka Connect in docker using a dedicated standalone properties file, set bootstrap servers to docker service names, and map plugin paths with docker volumes for seamless dockerized workers.

Prepare and copy JDBC source and sink connector properties for Docker Compose use, switching localhost to host docker internal so Kafka Connect runs in Docker and can reach host databases.

Run Kafka Connect in Docker compose standalone mode by using an environment file to pass variables such as connector plugins path, standalone config path, and JDBC source and sink configs.

Start Kafka Connect in standalone mode with a docker compose file and environment file to launch the Kafka cluster and MySQL and PostgreSQL databases.

Explore kafka connect distributed mode with multiple workers, using the rest api to manage connectors and rebalance tasks as you scale, with configuration and offsets stored in kafka topics.

In distributed mode, Kafka Connect creates internal topics, connect configs, connect offsets, and connect status, to track configurations, offsets, and connector statuses for multi-worker coordination and scalability.

Create a new connect distributed properties file for worker one by copying the default, then open it in Visual Studio Code to configure a kafka connect cluster in distributed mode.

Configure bootstrap servers for Kafka Connect in distributed mode to locate your Kafka brokers, using localhost:9092. In production, list multiple brokers to improve reliability if one broker fails.

Set the group ID to join Kafka Connect workers into a single cluster when running in distributed mode; in standalone mode this property isn’t required.

Configure key and value converters in Kafka Connect to translate Kafka bytes into internal formats, using JSON by default, with options to mix formats like string and Avro.

Understand how Kafka Connect uses offset storage topic and replication factor. Configure config storage topic and replication factor safeguards configurations, and status storage topic and replication factor monitor status.

Learn how to configure Kafka Connect rest api listeners and advertised endpoints, enabling http requests on port 8083 to manage connectors with curl, postman, or any http client.

Configure the Kafka Connect plugin path to tell where to scan for connectors, converters, and transformations. Add jars to the plugins folder, separate paths by comma, then restart the worker.

Duplicate the first worker's Kafka Connect config to create a second worker in distributed mode, keep the same group.id, and set rest api port to 8084.

Run Kafka Connect in distributed mode using local CLI scripts on macOS, with two workers sharing tasks on a three-broker cluster connected to MySQL as source and PostgreSQL as sink.

Deploy a JDBC source connector on a Kafka Connect cluster to stream data from the MySQL users.user_profiles table into Kafka topics using a provided JSON config.

Deploy a source connector to a distributed Kafka Connect cluster via the REST API. Send an HTTP POST to localhost:8083/connectors with the JSON configuration to start data flow into Kafka.

Review the JDBC sink connector for distributed mode to read from the Kafka topic users_user_profiles and write to the PostgreSQL database using the attached configuration.

Deploys a sync connector using the Kafka Connect rest api in distributed mode by posting a json configuration to the connectors endpoint on a running worker.

Demonstrates Kafka Connect distributed mode by verifying source and sink connectors transfer data from MySQL to a Kafka topic and into PostgreSQL, with hands-on Docker setup and console consumer validation.

Use the Kafka Connect rest api to list deployed connectors with the /connectors endpoint, then manage and monitor them via subsequent api calls (status, update, pause, resume).

Fetch and review the full configuration of a Kafka Connect connector using the connectors/{name} API and the config endpoint for a flat config object, enabling troubleshooting and updates.

Check a specific connector's status with the Kafka Connect rest api by issuing a get request to /connectors/{name}/status, confirming it runs and showing its tasks for a source connector.

Learn how to pause and resume a running Kafka Connect connector using the REST API, check status, and manage connectors with HTTP PUT requests.

Update a Kafka connector configuration via http put to the config endpoint with json. Verify by fetching the connector configuration and updating the topic prefix from users to app users.

Delete a Kafka Connect connector via the REST API. Verify that the connector and its tasks are removed without deleting Kafka data or databases.

Learn to run Kafka Connect in distributed mode with Docker, packaging all dependencies into one image for portable, scalable clusters, using Docker Compose to deploy three brokers and separate workers.

Learn to build a Docker image that packages Kafka Connect and required plugins, then run it as Docker containers on any server using a Dockerfile that defines the build steps.

Create a docker file to build a single Kafka Connect image with JDBC connectors, including MySQL and PostgreSQL drivers, using Confluent Hub and plugin path configuration.

Publish your locally built docker image to Docker Hub by tagging it with your repository name, then push and pull to run on other machines.

Create and use an environment file to configure Kafka Connect distributed mode in Docker, specifying bootstrap servers, group id, internal topics, converters, plugin path, and runtime rest api port.

Learn to run a Kafka Connect worker in distributed mode inside a Docker container using an environment file, advertised hostname and port, and rest api checks.

Run a second Kafka Connect worker in Docker by adjusting ports and container name, verify the rest API on port 8085 with Postman, and confirm the worker is up.

Deploy a JDBC source connector to a Kafka Connect cluster in distributed mode via rest API, using http post and host docker internal for MySQL connectivity.