Apache Kafka Crash Course for Java and Python Developers

Explore the basics of Kafka architecture, including topics, partitions, and event anatomy, and trace the data flow from producers to consumers in Kafka.

Explore how data flows from producers to Kafka and to consumers in a real-time publish-subscribe system, enabling loose coupling. Make messages immutable and store them in an append-only log.

Learn how Apache Kafka uses a cluster of brokers to store data written by producers and consumed by consumers, enabling scalable, fault-tolerant replication, with ZooKeeper gradually removed.

Apache Kafka uses topics and partitions to distribute writes across brokers for higher throughput, while replication and partition leaders provide fault tolerance and scalable consumption via consumer groups and offsets.

Decode the anatomy of a Kafka message—key, timestamp, offset, and value—and how producers set the key and payload while the same key routes to the same partition, serialized as bytes.

Explore the five open source Kafka APIs—admin, producer, consumer, connect, and streams—and learn how connectors simplify moving data between Kafka and systems like PostgreSQL, MongoDB, and S3.

Explore Apache Kafka API implementations across languages, including native Java admin, producer, consumer, and streams APIs, and librdkafka bindings for Python, C++, C#, Go, Swift, and Node.

Learn to manage Kafka topics with the Kafka topics CLI using the admin API, including listing, creating (with partitions and replication), describing, and deleting topics in a Docker Compose environment.

Kafka topics with advanced configurations, including per-topic retention and compacted topics, using topics and configs tools for create, describe, and alter operations.

Demonstrates creating a Kafka topic with the Python Kafka library and the Admin API, integrated into a FastAPI app within a dockerized environment, including handling existing topics.

Learn to create a Kafka topic on Spring Boot startup using the Java admin API, configuring bootstrap servers and topic settings (name, partitions, replicas) with a Spring Kafka config bean.

Observe how a Kafka producer maps messages to topic partitions using a key, achieving ordering within partitions, then serializes data to bytes and applies a hash-based partitioning algorithm.

Discover how producers map messages to partitions to boost Kafka scalability, using key-based hash-partitioning and round-robin when keys are absent.

Understand how Kafka tracks message consumption through consumer groups, offsets, and the consumer_offsets topic, including group IDs, partition assignments, and rebalance scenarios.

Explore how consumer group rebalances enable scalable, fault-tolerant consumption in Kafka by redistributing partitions among more consumers and tracking progress via the consumer offsets topic.

Build a basic Java Kafka consumer with Spring for Kafka, configure bootstrap servers and topics, define a consumer factory and Kafka listener, and log processed person messages.

Explore how the Kafka consumer flow manages offset commits and at least once processing, including default settings and rebalance behavior. See how default auto commit timing affects processing.

Disable auto commit and perform manual offset commits for at least once processing, risking possible reprocessing if failures occur, with batch commits in Spring Kafka and explicit commits in Python.

Disable auto commit and use manual offset commits for at most once processing, risking lost records for throughput and latency. Polling, batching, and committing the latest offset precede processing.

Develop an advanced Python Kafka consumer for exactly-once processing by disabling auto commit and manually committing offsets after each record, using topic partitions and offset metadata.

Learn to build an advanced Spring for Kafka Java consumer that commits after each record, achieving near exactly-once processing by enabling record mode and disabling auto commit.

Use the ECI tools to inspect and view consumer lag for a specific consumer group, listing topics, partitions, and offsets to reveal current lag.

Explore the Confluent Schema Registry, a Kafka metadata management tool for messages and their data structures, and learn how it governs schemas across Kafka topics.

Explore how the Confluent Schema Registry provides a RESTful interface for Kafka metadata and messages, supports JSON, Protobuf, and Avro, and enables safer schema evolution through configurable compatibility cards.

Explore how the Confluent Schema Registry integrates with a Kafka cluster to govern Avro-serialized data through a REST API, schema IDs, and runtime compatibility checks for schema evolution.

Explore Apache Avro, a core Kafka serialization technology, focusing on schema definitions, records, and fields in JSON or IDL, with optional fields and evolving schemas for producer and consumer applications.

Explore schema registry compatibility settings, including backwards, forwards, and transitive rules, and configure compatibility levels in Docker and via the rest API, with Avro integration for producers.

Build a Spring Kafka producer using Avro serialization and a schema registry, integrating the Avro domain events library, a REST endpoint, and dockerized schema registry for end-to-end Avro messaging.

build a spring boot kafka consumer integrated with avro and the schema registry, using a reusable avro domain events library and confluent deserializers to process messages.

Master Avro serialization and schema evolution with schema registry, learning backwards-compatible changes like adding optional fields and updating producers and consumers safely.

Explore the Confluent Schema Registry REST API to manage subjects for topic key or value schemas, query versions and IDs, validate compatibility, and register schemas via post requests.

Explore how Kafka Connect, a framework for ingesting data from external systems into Kafka and copying data out with source and sync connectors, runs on workers.

Discover why Kafka Connect provides a robust, scalable low-code data pipeline with Kafka as the intermediary, backed by a vast ecosystem of connectors and a flexible API for customization.

Explains how Kafka Connect fits into a Kafka-driven architecture, ingesting data from databases, flat files, and apps via connectors, and sinking to Elasticsearch, cloud storage, and analytics platforms.

Set up a local dockerized Kafka environment, build a custom Docker image with the Data Gen connector, and run Docker Compose up to view logs and learn Kafka Connect basics.

Query the Kafka Connect REST API to discover plugins. Configure and start a data gen connector with an Avro schema and schema registry, producing technologists data to a Kafka topic.

Demonstrates wiring a Kafka topic to MongoDB using the MongoDB Sync Connector in Kafka Connect, including Docker Compose setup, topic creation, and verifying data flow into MongoDB.

Explore the Kafka streams API, shipped with Apache Kafka, to build scalable, stream processing apps over event streams using the DSL or processor API for event-driven microservices and real-time analytics.

Learn stateless and stateful transformations in stream processing, applying map and filter to compute revenue per event, then group by customer ID and perform aggregates like reduce and windowing.

Explore stream processing topologies built from source, sink, and stateful or stateless transforms forming directed acyclic flow. See order validation and revenue topologies that join and enrich data for insights.

Kafka streams rely on topics and partitions. Topologies run as tasks in application instances, with each partition consumed by one task, so adding instances boosts throughput.

Build a multi-processor Kafka Streams app that enriches orders with product data, computes per-customer revenue, and outputs enriched results to a revenue topic.

Learn how sliding windows provide real-time insights over a 60-second interval, with updates every 20 seconds. Compare sliding windows to tumbling ones and note potential double-counting.

Explore stream processing with Python using the fast Python stream processing library in this section of the Apache Kafka crash course for Java and Python developers.

Faust is a Python based stream processing library that ports Kafka Streams concepts to Python, using dictionaries and async/await instead of a DSL, enabling event-driven microservices and AI/ML predictions.

Demonstrate Faust tasks and timers by running a demo app that logs a startup task and a timer that fires every 10 seconds.

Demonstrates a Faust producer and consumer using a timer to emit random greetings to a Kafka topic every five seconds, with docker-managed services for end-to-end streaming.

Learn to process complex types in Faust by defining a Faust record schema for message keys and values, creating greeting objects, and producing and consuming typed messages.

Produce and consume on the Kafka greetings topic with a complex greeting object, sending the greeting record as the value and deserializing back on consumption.

Explore Faust tables as distributed data structures, backed by a change log topic for fault tolerance and reassembly, with group by greeter and co-partitioning for coherent aggregation.

Demonstrates building an aggregate table with Faust tables by consuming greetings events, grouping by the greeter, and incrementing per-greeter counts to produce a running total.