Kafka fundamentals for java developers

Capture and store microservice events as a distributed commit log with replication to prevent data loss. Enable real-time analytics using Kafka's streaming API for grouping, aggregating, filtering, and joining.

Discover why Kafka is popular for its disk-based persistence, scalability, availability, multi-producer and multi-consumer support, topic partitions, and consumer groups that enable parallel processing within a group while preserving messages.

Explore how Kafka enables messaging across microservices, supports producer–consumer patterns, tracks user activity, aggregates metrics and logs, and powers stream processing and data pipelines.

Explore how Kafka underpins big data platforms and microservices, enabling real-time tracking of IoT devices from factories to shipments, and powering financial, healthcare, and customer experience use cases.

Kafka powers real-time analytics and streaming for major apps like Twitter, Uber, Netflix, Yahoo, and Pinterest, handling billions of events daily and supporting log aggregation and real-time advertising.

Explore how Kafka uses brokers, ZooKeeper, producers, and consumers to enable durable messaging. Cluster leaders manage partitions and replicas; producers publish to topics, and consumers coordinate progress.

Learn how a Kafka record is produced, including its seven attributes: topic, partition, offset, timestamp, key, headers, and value, and how these fields determine partitioning and data routing.

Learn how Kafka topics are divided into partitions and offsets, how messages are appended to partitions, and how replication and leader-follower dynamics ensure scalability and high availability.

Learn how Kafka partitions enable concurrency by forming consumer groups, where each partition is owned by a single consumer, enabling horizontal scaling and coordination when a consumer fails.

Learn how Kafka producers batch messages by topic and partition to reduce network round trips, with compression for efficient transfer, while configuring batch size to optimize delay and processing time.

Set up Kafka on your local machine with Windows and Mac, use the command line to create and manage topics, and run producers and consumers to send and receive messages.

Choose the Java version for this course: Java 11 or the latest with Kafka in standalone mode; Confluent platform supports Java 8–11, so switch to Java 11 for schema registry.

Learn to use the Kafka command line to list, create, describe, and delete topics, and to produce and consume messages with console tools via a bootstrap server.

Explore Kafka's five core APIs—admin API, producer API, consumer API, streaming API, and connect API—that enable cluster management, message publication, stream processing, and data integration with external systems.

Explore Kafka fundamentals through official documentation from the Apache Kafka site and Confluent docs to learn producer and consumer configurations, APIs, and the schema registry.

Explore the Kafka producer workflow, from record creation to broker delivery, highlighting topics, keys, values, headers, Avro serialization, partitioning, batching, and retry logic.

Learn to create a Kafka producer, configure mandatory properties (bootstrap servers, key and value serializers), and publish a record to a topic using producer.send with fire-and-forget, synchronous, or asynchronous callbacks.

Create a Kafka producer project using Maven in Spring Tool Suite, selecting the Maven archetype quick start, and add the Kafka clients dependency to the pom.xml.

Create a Java Kafka producer by configuring bootstrap servers, key serializer, and value serializer, then send an order record to the order topic and confirm on success.

Send a message to a kafka topic synchronously by waiting for the future's record metadata, including partition and offset. Large volumes may add delays as the response arrives.

Learn to implement a synchronous and asynchronous send in Kafka by using a send method overload with a callback, handle on completion, retrieve partition and offset, and manage exceptions.

Configure a Kafka consumer with bootstrap servers, group id, and key/value deserializers, then subscribe to topics and poll for records.

Create a Maven-based consumer project for Kafka by scaffolding with an archetype, configuring the pom, adding Kafka clients, and syncing the project in the Spring Tool Suite.

Create a Kafka consumer, configure bootstrap servers, key and value deserializers, and a group ID; subscribe to topics and poll for records, then print product name and quantity.

Learn to create custom Kafka serializers and deserializers for Java objects, converting objects to byte arrays with Jackson's object mapper, and wire them into producers and consumers.

Create a custom serializer for a Java pojo using Jackson to convert the order object to bytes, and configure the Kafka producer to use this serializer for a dedicated topic.

Create a custom deserializer for Kafka by implementing the Apache Kafka deserializer, using Jackson's ObjectMapper to convert byte arrays to Order objects, and update the producer and consumer.

Discover how Apache Avro simplifies Kafka data exchange by providing out-of-the-box serializers and deserializers driven by a JSON schema, with a schema registry coordinating producer and consumer.

Explain how Avro schemas evolve and how the schema registry manages multiple versions for downstream consumers and deserializers, using Confluent's open source registry.

Explore how Avro serializers and deserializers work with the schema registry to manage schemas, subjects, versioning, and data in Kafka, including header-embedded schema ids and producer-consumer interactions.

Learn to set up the Confluent schema registry for Kafka, run the Confluent platform locally, and manage Avro schemas with the registry and the command line interface.

Create an Avro schema for the order use case by adding order.avsc under src/resources, defining a record Order with fields customer name, product, and quantity; set namespace com.bharath.kafka.avro.

Walk through setting up kafka avro in java: add maven dependencies, generate order pojo from schema, and use the kafka avro serializer with a schema registry for producers and consumers.

Configure the Maven Avro plugin to generate Java classes from your Avro schema, pulling dependencies from confluent, and outputting generated sources for use with Kafka Avro serialization.

Create a Kafka producer that uses Avro serializer to serialize a POJO, set schema registry URL, and publish to Avro topic Order Avro, converting the customer name to string.

Create a Kafka consumer using Avro deserializers, configure the schema registry URL and specific Avro reader, and use a shared generated Avro stub for the order Avro topic.

Explore how Avro serializers and deserializers interact with the schema registry to serialize the order object, publish its value schema, and validate data in producer and consumer flows.

Apply Avro generic records to Kafka by creating a schema, dynamically building a generic record that complies with it, and sending it with the serializer and schema registry.

Create a kafka producer that uses avro generic records, parsing an avro schema to dynamically build a generic record and send it to a generic avro topic.

Create a Kafka consumer that processes generic Avro records, extracting product and quantity from the generic record, using Avro deserializers and the schema registry.

Explore schema evolution, how the schema registry preserves versions and assigns ids in headers, and how producers and consumers handle field additions or removals with avro deserializers to maintain compatibility.

Explore how evolving schemas affect downstream consumer applications and enforce backward compatible changes with a schema registry and producer owned schemas. Define rules to prevent breaking changes and ensure compatibility.

Understand how the Kafka consumer polls for records in real time, coordinates partitions and heartbeats, and runs in an infinite loop with a safe finally close.

Stop Confluent services to switch from Confluent Kafka and Schema Registry to your standalone Kafka, then restart ZooKeeper and Kafka as before; otherwise topics will be created on Confluent brokers.

learn how kafka partitioners assign records using an explicit partition or key-based hash, create a custom partition, and explore advanced producer configurations and broker reactions.

Create a topic with multiple partitions by manually configuring Kafka topics, describe topics to verify partitions, and prepare producer and consumer code with custom serializer and deserializer for future partitioning.

Create a custom partitioner for the Kafka producer and implement the partition logic, including a custom route for a specific key, using Murmur2 hashing and cluster partition info.

Explore the Kafka producer configuration with the ProducerConfig class, covering the mandatory properties, including bootstrap servers and key and value serializer configs.

Learn how producer acks settings (0, 1, all) affect durability, and how buffer memory, compression type (snappy, gzip, and lz fuor), and retries with backoff influence delivery latency.

Explore three more important Kafka producer configuration properties: batch size (memory in bytes), linger milliseconds, and request timeout ms, and see how they affect throughput and latency.

Explore the three Kafka producer delivery semantics, including at-least-once and idempotency to prevent duplicates. Learn four steps to enable producer transactions like JDBC transactions.

Explore the three delivery semantics—at least once, at most once, and exactly once—covering retries, acknowledgement, and how idempotence prevents duplicates.

Explore how Kafka producer transactions ensure atomicity from init transactions through begin transaction to commit or abort, with rollback and partition handling.

Learn to use kafka producer transactions to ensure atomic writes by assigning a unique transactional id, initializing transactions, and wrapping sends with begin, commit, and abort.

Understand that a single producer cannot open multiple transactions simultaneously; start the transaction before any threads send, then commit after all finish, flushing unsent records, and abort on errors.

Revisit consumer group rebalancing and its impact on groups, partitions, and offsets. Learn to minimize rebalancing risk with offset commits, including synchronous, asynchronous, and specific-offset commits, then explore advanced consumer configurations.

Explore how consumer groups enable scalable, parallel consumption from Kafka topics by partitioning data and distributing load across multiple consumers, with billing and housekeeping apps each getting full data copies.

Learn how consumer group rebalancing in Kafka reallocates partitions when a consumer joins or leaves, enabling high availability and scalability, while noting the brief silence during rebalances.

Understand how Kafka consumer groups use the group coordinator and a group leader to rebalance partitions as members join or leave, driven by heartbeats and health checks.

Learn how offset commits enable tracking across partitions, how rebalance uses the __consumer_offsets topic to resume, and how autocommit and manual commits prevent duplicate or missed processing.

Understand auto commit in Kafka consumers, disable with enable.auto.commit=false, adjust timing via auto.commit.interval.ms, and note that poll drives commits, close also commits, and rebalance can cause duplicates if uncommitted.

Disable auto commit and manually manage offset commits in Kafka consumers; process records after poll, then use sync commit to ensure committed offsets, mitigating rebalancing duplicates.

Commit async processes the next offsets without blocking the consumer, but it does not retry on failure and may cause duplicates after rebalance; use the commit callback for logging or reprocessing.

Implement custom commits with the async method to commit offsets every ten records, using a topic-partition map and offset plus one to handle rebalances.

Create a rebalance listener by implementing the consumer rebalance listener in an inner class, handling on partitions revoked and on partitions assigned, and pass it to subscribe to commit offsets.

Learn how to commit uncommitted offsets during a rebalance by maintaining a current offsets map and using commit sync to commit the latest processed offsets before partitions are revoked.

Configure the Kafka consumer using the ConsumerConfig class constants, replace hard-coded values, and set bootstrap servers, key deserializer, value deserializer, and group id config.

tune fetch min bytes and fetch max wait to control data delivery, and set heartbeat interval as about one third of the session timeout to prevent rebalance.

Learn four more Kafka consumer properties: max partition fetch bytes, auto offset reset (latest or earliest), client id, and max poll records, with defaults and impact on partition handling.

Configure partition assignment strategy via consumer config to choose between the default range assigner and the round robin assigner, balancing partitions across consumers.

Learn to create standalone Kafka consumers that bypass group coordination, use the assign method to allocate partitions, and monitor partitions using the partitions for method.

Create a simple Kafka consumer by using consumer.assign to bind to all partitions of a topic, building topic partitions, and processing records from the simple consumer topic.

Create a simple producer and test a standalone consumer by configuring a separate topic, running the producer first to publish messages, and using offset reset to start from the earliest.

Learn how simple consumers poll batches, rely on default auto commits, and why setting a unique group.id enables manual offset commits and avoids group-related exceptions.