Apache Kafka Crash Course for Java and Python Developers

Explore the basics of Kafka architecture, including topics, partitions, and event anatomy, and trace the data flow from producers to consumers in Kafka.

Discover how Apache Kafka enables event driven architectures with state transfer and receiver driven routing, via an order created event flowing through a Kafka topic to invoicing, shipping, and commissions.

Explore how data flows from producers to Kafka and to consumers in a real-time publish-subscribe system, enabling loose coupling. Make messages immutable and store them in an append-only log.

Learn how Apache Kafka uses a cluster of brokers to store data written by producers and consumed by consumers, enabling scalable, fault-tolerant replication, with ZooKeeper gradually removed.

Apache Kafka uses topics and partitions to distribute writes across brokers for higher throughput, while replication and partition leaders provide fault tolerance and scalable consumption via consumer groups and offsets.

Decode the anatomy of a Kafka message—key, timestamp, offset, and value—and how producers set the key and payload while the same key routes to the same partition, serialized as bytes.

Discover how kafka stores messages as append-only logs in topic partitions, with default seven-day retention via retention.ms and keys determining partition routing for a chronological event history.

Explore the five open source Kafka APIs—admin, producer, consumer, connect, and streams—and learn how connectors simplify moving data between Kafka and systems like PostgreSQL, MongoDB, and S3.

Explore Apache Kafka API implementations across languages, including native Java admin, producer, consumer, and streams APIs, and librdkafka bindings for Python, C++, C#, Go, Swift, and Node.

Discover the Admin API to inspect and manage Kafka storage architecture components, including brokers, topics, and partitions, designed for Java and Python developers.

Learn to manage Kafka topics with the Kafka topics CLI using the admin API, including listing, creating (with partitions and replication), describing, and deleting topics in a Docker Compose environment.

Kafka topics with advanced configurations, including per-topic retention and compacted topics, using topics and configs tools for create, describe, and alter operations.

Demonstrates creating a Kafka topic with the Python Kafka library and the Admin API, integrated into a FastAPI app within a dockerized environment, including handling existing topics.

Use the Python admin client API to create topics with nondefault config values, including retention milliseconds set to 360,000, and alter existing topics on startup in a fast API app.

Learn to create a Kafka topic on Spring Boot startup using the Java admin API, configuring bootstrap servers and topic settings (name, partitions, replicas) with a Spring Kafka config bean.

Explore the Kafka Producer API and learn how to write events into Apache Kafka, enabling decoupled integration with downstream consumer applications.

Observe how a Kafka producer maps messages to topic partitions using a key, achieving ordering within partitions, then serializes data to bytes and applies a hash-based partitioning algorithm.

Discover how producers map messages to partitions to boost Kafka scalability, using key-based hash-partitioning and round-robin when keys are absent.

Learn to produce data into Kafka with console producer and verify with console consumer in a dockerized seal tools container, showing print and parse keys on people topic.

Build a Python Kafka producer behind a FastAPI rest endpoint that creates fictitious people objects and publishes them to a Kafka topic.

Builds a Java producer in Spring for Kafka that publishes person objects to a rest endpoint using a Kafka template, with YAML config for key and JSON value serialization.

Adopt a code-based Spring for Kafka configuration using a config class, producer factory, and Kafka template to avoid typos and use IDE support. Verify by producing ten records.

Explore how the Kafka producer works under the hood, tune reliability and performance with batching, acks, retries, delivery timeout, idempotence, and partitioning strategies.

Explains how increasing Kafka topic partitions boosts throughput but can break key-based ordering due to the hash modulo partition calculation, highlighting the trade-off between scalability and ordering.

Expand a basic Python producer by tuning acks all, max in-flight requests to one, and retries; implement success and error callbacks and logging for reliability.

Explore building an advanced Java Kafka producer, configuring robust settings (acks, idempotence, max in-flight, retries, batch size, linger), creating advanced topics, and using callbacks for publish results.

Understand how Kafka tracks message consumption through consumer groups, offsets, and the consumer_offsets topic, including group IDs, partition assignments, and rebalance scenarios.

Explore how consumer group rebalances enable scalable, fault-tolerant consumption in Kafka by redistributing partitions among more consumers and tracking progress via the consumer offsets topic.

Build a basic Java Kafka consumer with Spring for Kafka, configure bootstrap servers and topics, define a consumer factory and Kafka listener, and log processed person messages.

Explore how the Kafka consumer flow manages offset commits and at least once processing, including default settings and rebalance behavior. See how default auto commit timing affects processing.

Disable auto commit and perform manual offset commits for at least once processing, risking possible reprocessing if failures occur, with batch commits in Spring Kafka and explicit commits in Python.

Disable auto commit and use manual offset commits for at most once processing, risking lost records for throughput and latency. Polling, batching, and committing the latest offset precede processing.

Develop an advanced Python Kafka consumer for exactly-once processing by disabling auto commit and manually committing offsets after each record, using topic partitions and offset metadata.

Learn to build an advanced Spring for Kafka Java consumer that commits after each record, achieving near exactly-once processing by enabling record mode and disabling auto commit.

Use the ECI tools to inspect and view consumer lag for a specific consumer group, listing topics, partitions, and offsets to reveal current lag.

Reset consumer group offsets for a topic partition using a dry run, then execute, stop consumers, and observe lag changes from earliest to latest with ECI tools.

Explore the Confluent Schema Registry, a Kafka metadata management tool for messages and their data structures, and learn how it governs schemas across Kafka topics.

Explore how the Confluent Schema Registry provides a RESTful interface for Kafka metadata and messages, supports JSON, Protobuf, and Avro, and enables safer schema evolution through configurable compatibility cards.

Use Confluent Schema Registry to manage data coupling through schema evolution and data exchange contracts, protecting downstream systems and enabling reliable integration with Apache Kafka.

Explore how the Confluent Schema Registry integrates with a Kafka cluster to govern Avro-serialized data through a REST API, schema IDs, and runtime compatibility checks for schema evolution.

Explore Apache Avro, a core Kafka serialization technology, focusing on schema definitions, records, and fields in JSON or IDL, with optional fields and evolving schemas for producer and consumer applications.

Explore schema registry compatibility settings, including backwards, forwards, and transitive rules, and configure compatibility levels in Docker and via the rest API, with Avro integration for producers.

Set up a Gradle Avro library to enable Avro serialization for producer and consumer, and generate Avro classes from a person schema for the schema registry jar.

Build a Spring Kafka producer using Avro serialization and a schema registry, integrating the Avro domain events library, a REST endpoint, and dockerized schema registry for end-to-end Avro messaging.

build a spring boot kafka consumer integrated with avro and the schema registry, using a reusable avro domain events library and confluent deserializers to process messages.

Master Avro serialization and schema evolution with schema registry, learning backwards-compatible changes like adding optional fields and updating producers and consumers safely.

Build a Python-based Kafka producer using Avro with Schema Registry, including topic creation, Avro serialization, FastAPI integration, and schema evolution in a Confluent environment.

Demonstrate backwards-compatible schema evolution in a python producer and consumer app integrated with schema registry, updating v1 to v2 with first name and last name, and validating with rest api.

Explore the Confluent Schema Registry REST API to manage subjects for topic key or value schemas, query versions and IDs, validate compatibility, and register schemas via post requests.

Explore how Kafka Connect, a framework for ingesting data from external systems into Kafka and copying data out with source and sync connectors, runs on workers.

Discover why Kafka Connect provides a robust, scalable low-code data pipeline with Kafka as the intermediary, backed by a vast ecosystem of connectors and a flexible API for customization.

Explains how Kafka Connect fits into a Kafka-driven architecture, ingesting data from databases, flat files, and apps via connectors, and sinking to Elasticsearch, cloud storage, and analytics platforms.

Learn to set up a separate Kafka Connect cluster, install connectors, and manage them via the rest api with put requests to configure topics, connector class, and tasks.

Set up a local dockerized Kafka environment, build a custom Docker image with the Data Gen connector, and run Docker Compose up to view logs and learn Kafka Connect basics.

Query the Kafka Connect REST API to discover plugins. Configure and start a data gen connector with an Avro schema and schema registry, producing technologists data to a Kafka topic.

Use the Kafka Connect REST API to manage a datagen connector: check status, pause, resume, and delete the connector, observing worker IDs and task states.

Demonstrates wiring a Kafka topic to MongoDB using the MongoDB Sync Connector in Kafka Connect, including Docker Compose setup, topic creation, and verifying data flow into MongoDB.

Explore the Kafka streams API, shipped with Apache Kafka, to build scalable, stream processing apps over event streams using the DSL or processor API for event-driven microservices and real-time analytics.

Learn two core Kafka streams DSL concepts: streams and tables. Streams model an immutable sequence of keyed events on a Kafka topic, while tables capture a grouped snapshot of state.

Learn stateless and stateful transformations in stream processing, applying map and filter to compute revenue per event, then group by customer ID and perform aggregates like reduce and windowing.

Explore stream processing topologies built from source, sink, and stateful or stateless transforms forming directed acyclic flow. See order validation and revenue topologies that join and enrich data for insights.

Kafka streams rely on topics and partitions. Topologies run as tasks in application instances, with each partition consumed by one task, so adding instances boosts throughput.

Demonstrate a Kafka Streams topology that consumes order created events, applies a 95/5 valid flag, and writes results to the order validated topic using Avro and Schema Registry.

Build a multi-processor Kafka Streams app that enriches orders with product data, computes per-customer revenue, and outputs enriched results to a revenue topic.

Explore tumbling windows in stream processing, non-overlapping 60-second blocks that bound a unique set of events. Learn how to calculate metrics like total revenue over each window.

Learn how sliding windows provide real-time insights over a 60-second interval, with updates every 20 seconds. Compare sliding windows to tumbling ones and note potential double-counting.

Explore stream processing with Python using the fast Python stream processing library in this section of the Apache Kafka crash course for Java and Python developers.

Faust is a Python based stream processing library that ports Kafka Streams concepts to Python, using dictionaries and async/await instead of a DSL, enabling event-driven microservices and AI/ML predictions.

Discover Faust library stream and table constructs as abstractions over Kafka topics, where streams are unbounded sequences of keyed events and tables are time-bounded snapshots of state.

Explore stateless and stateful streaming computations in Apache Kafka, mastering map, filter, group-by aggregation, and windowing to compute revenue per customer.

Discover the Faust programming model with agents that process unbounded event streams, using in-memory or Kafka-backed channels, topics, and infinite streams to drive real-time applications.

Install Faust and set up a Python 3.8 environment, then spin up a local Kafka cluster with Docker. Run the agents demo to process greetings messages in real time.

Explore Faust tasks and timers in the Fosse library by implementing a startup task that runs once and a timer that runs every 10 seconds, printing the date and time.

Demonstrate Faust tasks and timers by running a demo app that logs a startup task and a timer that fires every 10 seconds.

Learn to produce to and consume from Kafka topics with Faust. Send greetings to the greetings topic and use a Faust agent to process and print each greeting.

Demonstrates a Faust producer and consumer using a timer to emit random greetings to a Kafka topic every five seconds, with docker-managed services for end-to-end streaming.

Learn to process complex types in Faust by defining a Faust record schema for message keys and values, creating greeting objects, and producing and consuming typed messages.

Produce and consume on the Kafka greetings topic with a complex greeting object, sending the greeting record as the value and deserializing back on consumption.

Explore Faust tables as distributed data structures, backed by a change log topic for fault tolerance and reassembly, with group by greeter and co-partitioning for coherent aggregation.

Demonstrates building an aggregate table with Faust tables by consuming greetings events, grouping by the greeter, and incrementing per-greeter counts to produce a running total.

Understand how Kafka topic partitions dictate the horizontal scalability of Faust streams apps, via topology, agents, and application instances, with parallelism capped by source topic partitions.