Hands-on Kafka Connect: Source to Sink in S3, GCS & Beyond

The Amazon S3 sink connector pulls data from Kafka partitions and writes each as an S3 object, supporting Avro, JSON, bytes, and string formats with exactly-once delivery and dead-letter queue.

Set up a docker kafka environment with lensesio fast-data-dev (zookeeper, broker, schema registry) and a kafka UI, then run a producer to publish dummy data for the s3 sink connector.

Learn to configure and run a standalone S3 sink connector for Kafka Connect, including topic setup, partitions, JSON data, and flush size tuning.

Start a distributed Kafka Connect worker and configure the bootstrap server and group ID. Manage S3 sink connector via REST API, monitor health, and update or delete connectors as needed.

Explore time-based partitioner with the s3 sink connector to partition by record time or system time, using flush.size, rotateIntervalMillisecond, and path format year month day hour minute.

Apply the field partitioner to route Avro records by the city field, creating city-based partitions in an S3 sink, while using Avro schema and schema registry with a producer.

Learn how the S3 sink connector handles schema evolution with Avro and schema registry, enabling backward compatibility as new fields appear in Kafka records.

Run a Kafka Connect worker in Docker in distributed mode, build and deploy an S3 sink connector image, configure with environment variables, and verify data flows from Kafka to S3.

Implement dead letter queue in s3 sink connector by routing invalid records to a dlq topic; configure errors.tolerance to none or all, and create the dlq topic before starting.

Automate your Kafka Connect workflow with Docker Compose to replace manual Docker run steps, build a reusable S3 sink image, and start all services with one command.

Set up a two-node Kafka Connect cluster with Docker and Docker Compose, deploying S3 sink workers on both nodes to test load balancing and fault tolerance.

Test task distribution and fault tolerance in a two-node Kafka Connect cluster by creating a 3-partition topic and validating S3 sink connector tasks on node 1 and node 2.

Explore Debezium MySQL cdc source connector, capturing data and schema changes and pushing events to Apache Kafka, including snapshot stage and ddl/dml observation.

Explore how the wcmysql connector uses binlog to capture and stream changes from snapshot to CDC via Debezium, with Kafka topics and the schema history topic.

Install and configure MySQL 8.0 for Debezium in a Kafka Connect pipeline, create Debezium user with read access to inventory, set binlog options, and build a test schema.

Launch Kafka, Kafka Connect, and MySQL via docker-compose, register the Debezium MySQL source connector with REST APIs in distributed mode on 8083, and monitor topics such as schema history inventory.

Explore how Debezium captures insert, update, and delete events from a MySQL database, including before/after payloads and binlog-based source data, and streams them to Kafka in CDC mode.

Demonstrates DDL statements such as alter and drop on MySQL, and how Debezium writes schema history into a single inventory Kafka topic, tracking snapshot and CDC events.

Explore integrating Debezium MySQL with schema registry to enforce Avro payloads, manage schema versions, and observe schema evolution across Kafka topics using Avro converter.