Apache Kafka Crash Course for Java and Python Developers

Explore the basics of Kafka architecture, including topics, partitions, and event anatomy, and trace the data flow from producers to consumers in Kafka.

Explore how data flows from producers to Kafka and to consumers in a real-time publish-subscribe system, enabling loose coupling. Make messages immutable and store them in an append-only log.

Apache Kafka uses topics and partitions to distribute writes across brokers for higher throughput, while replication and partition leaders provide fault tolerance and scalable consumption via consumer groups and offsets.

Explore the five open source Kafka APIs—admin, producer, consumer, connect, and streams—and learn how connectors simplify moving data between Kafka and systems like PostgreSQL, MongoDB, and S3.

Explore Apache Kafka API implementations across languages, including native Java admin, producer, consumer, and streams APIs, and librdkafka bindings for Python, C++, C#, Go, Swift, and Node.

Learn to manage Kafka topics with the Kafka topics CLI using the admin API, including listing, creating (with partitions and replication), describing, and deleting topics in a Docker Compose environment.

Discover how producers map messages to partitions to boost Kafka scalability, using key-based hash-partitioning and round-robin when keys are absent.

Explore how consumer group rebalances enable scalable, fault-tolerant consumption in Kafka by redistributing partitions among more consumers and tracking progress via the consumer offsets topic.

Disable auto commit and perform manual offset commits for at least once processing, risking possible reprocessing if failures occur, with batch commits in Spring Kafka and explicit commits in Python.

Learn to build an advanced Spring for Kafka Java consumer that commits after each record, achieving near exactly-once processing by enabling record mode and disabling auto commit.

Explore the Confluent Schema Registry, a Kafka metadata management tool for messages and their data structures, and learn how it governs schemas across Kafka topics.

Explore schema registry compatibility settings, including backwards, forwards, and transitive rules, and configure compatibility levels in Docker and via the rest API, with Avro integration for producers.

build a spring boot kafka consumer integrated with avro and the schema registry, using a reusable avro domain events library and confluent deserializers to process messages.

Explore the Confluent Schema Registry REST API to manage subjects for topic key or value schemas, query versions and IDs, validate compatibility, and register schemas via post requests.

Discover why Kafka Connect provides a robust, scalable low-code data pipeline with Kafka as the intermediary, backed by a vast ecosystem of connectors and a flexible API for customization.

Explains how Kafka Connect fits into a Kafka-driven architecture, ingesting data from databases, flat files, and apps via connectors, and sinking to Elasticsearch, cloud storage, and analytics platforms.

Set up a local dockerized Kafka environment, build a custom Docker image with the Data Gen connector, and run Docker Compose up to view logs and learn Kafka Connect basics.

Query the Kafka Connect REST API to discover plugins. Configure and start a data gen connector with an Avro schema and schema registry, producing technologists data to a Kafka topic.

Demonstrates wiring a Kafka topic to MongoDB using the MongoDB Sync Connector in Kafka Connect, including Docker Compose setup, topic creation, and verifying data flow into MongoDB.

Learn stateless and stateful transformations in stream processing, applying map and filter to compute revenue per event, then group by customer ID and perform aggregates like reduce and windowing.

Explore stream processing topologies built from source, sink, and stateful or stateless transforms forming directed acyclic flow. See order validation and revenue topologies that join and enrich data for insights.

Build a multi-processor Kafka Streams app that enriches orders with product data, computes per-customer revenue, and outputs enriched results to a revenue topic.

Demonstrate Faust tasks and timers by running a demo app that logs a startup task and a timer that fires every 10 seconds.

Learn to process complex types in Faust by defining a Faust record schema for message keys and values, creating greeting objects, and producing and consuming typed messages.

Produce and consume on the Kafka greetings topic with a complex greeting object, sending the greeting record as the value and deserializing back on consumption.

Demonstrates building an aggregate table with Faust tables by consuming greetings events, grouping by the greeter, and incrementing per-greeter counts to produce a running total.