Apache Kafka for developers

Explore Kafka concepts, use cases across industries, the core data model of messages and topics, and the producers, consumers, and brokers in distributed, fault-tolerant Kafka architecture, with hands-on topic creation.

Explore how Kafka ingests streams from thousands of sources for real-time analytics. See how data is stored in Hadoop or Cassandra, processed by Spark, and visualized in Tableau.

Learn how Kafka treats every data item as a message or record with a key, value, and timestamp, and how messages are organized into topics like logrecords and salesdata.

Learn how producers and consumers interact with Kafka topics, using console producer and console consumer for testing, and Java APIs to produce and consume messages.

Kafka brokers are processes in a multi-node cluster that receive messages from producers, store them in topics, and deliver messages to consumers on request while tracking each consumer's progress.

Serialization and deserialization, or ser-de, convert program objects to and from a stream of bytes, with Kafka serializers for common types like string, integer, long, double, and bytes.

Learn how Kafka enables distributed processing by partitioning topics, allowing parallel producers and consumers; messages are ordered within each partition and consumed by dedicated consumers for horizontal scaling.

Explore zookeeper, an open source tool for distributed process coordination that enables leader selection, exclusive locks, and robust interprocess communication to prevent partial failures and race conditions.

Explore how Apache Kafka achieves fault tolerance by persisting messages on partitions to disk, replicating them across brokers, and electing leaders through Zookeeper.

Discover how Kafka brokers manage partitions, with producers sending messages, consumers reading them in consumer groups, with multiple groups existing, and Zookeeper coordinating the process.

create topics with the kafka-topics cli by specifying a name, partitions, and replication factor, connecting to zookeeper at localhost:2181 and ensuring replication factor does not exceed broker count.

Master the kafka-topics command to list topics and create new topics with partitions and replication factors, and learn how replication factor must not exceed available brokers.

Learn to send and read messages in Kafka cluster without coding, using kafka-console-producer and kafka-console-consumer. Connect to localhost:9092 with bootstrap-server and topic newtopic1 to see messages appear in real time.

Demonstrate creating and reading messages with keys and values using the Kafka console producer and consumer, configuring key separators and print key options on topic newtopic1.

Learn to use Apache Kafka for industry message processing, create topics with keys and values, read messages with the consumer, and review installation and configuration of Zookeeper and Kafka brokers.

Choose suitable hardware and operating system for Kafka installation, select a Kafka version, and install the software; start Kafka and Zookeeper with minimal configuration while reviewing essential development options.

Plan hardware and operating system choices for Kafka deployments, detailing development and production memory, CPU, storage requirements, and zookeeper fault tolerance.

Choose the latest stable kafka version for installation from kafka.apache.org or confluen.io, and use confluent 5.1 for this class; configure kafka-clients in pom.xml with group org.apache.kafka and artifact kafka-clients.

Learn the prerequisite software for Kafka: install Java JDK 1.7 or higher from Oracle or open source, and use Zookeeper for process coordination, bundled or separately.

Install and run Apache Kafka on Windows using the Confluent distribution, explore the directory structure, and start ZooKeeper and Kafka server with the provided properties files.

Install kafka on linux from a tarball into /usr/local/kafka, set JAVA_HOME and PATH, and start zookeeper and kafka in the background with nohup; stop using the stop scripts.

Install Apache Kafka 2.2.0 on Ubuntu 16, extract the tarball to /usr/local/kafka, configure the path, start zookeeper and Kafka servers with nohup, manage topics, and stop the services.

Explore Kafka broker configurations from server.properties, including zookeeper.connect, broker.id, log.dirs, log.retention.hours, and listeners such as plaintext on port 9092.

Fix startup failures on Windows by clearing all log files in Kafka-logs, then restart the server; if it aborts again, reinstall Kafka, not for production.

Install and start ZooKeeper and Kafka servers across Windows, open source, and Confluent options, and compare development and production configurations. Review the ZooKeeper bundle and broker configuration options.

Install Eclipse to develop Kafka programs in Java, utilize Maven for library dependencies in pom.xml, and run Kafka client programs from Eclipse to connect to any Kafka cluster.

Set up Eclipse for Kafka development by configuring the workspace, adding the JRE system library (JDK 1.8), adjusting compiler settings to 1.7, and building the Kafka_prodSimple project.

Open and inspect the pom.xml in kafka_prodSimple to manage maven dependencies, including kafka version 2.1.0, kafka clients with provided scope, log4j libraries, and jdk.tools path, enabling compile-time Kafka programs.

Import kafka projects into Eclipse using Maven, selecting existing Maven projects and the directory C:eclipse_kafka. Then finish and configure the compiler and build path to view all projects.

Set up the eclipse run configuration to execute a java application, selecting run_kafka_prodSimple in the kafka_prodSimple project and specifying two vm arguments: -Dkafka.logs.dir=C:/kafka/logs and -Dlog4j.configuration=file:///C:/kafka/etc/kafka/log4j.properties.

Master Kafka producers by building and running producer programs, using callbacks and synchronous processing, exploring keys, values, serializers, and custom partitioning logic.

Add messages to a Kafka topic by sending them to the broker. Target one or more partitions, serialize data before sending, and preserve order as received.

Use the Kafka producer API to publish messages to a topic by configuring a Java properties object, setting broker address, batch size, serializers, then create and close the producer.

Configure a java kafka producer with localhost:9092 and string serializers, then send 20 messages to topic mytopic1 with keys key_1 to key_20 and values equal to i squared, and close.

Run a simple producer using the kafka_prodSimple run configuration, specifying vm arguments for logs and log4j properties, then observe produced messages with a kafka console consumer on localhost:9092, topic mytopic1.

Explore the producer API from top to bottom, configuring properties and importing producer libraries in Apache Kafka. Inspect key properties and available client libraries to set up the producer effectively.

Explore the key producer configuration properties such as bootstrap.servers, key/value serializers, acks, buffer.memory, batch.size, linger.ms, and partitioner.class to understand how batching, fault tolerance, and serialization shape Kafka producer behavior.

Explore configuring a Kafka producer in eclipse by adjusting bootstrap.servers, acks, batch.size, linger.ms, and buffer.memory; review serializers and producer records for multiple topics, and use asynchronous sends with futures.

The producer send API is asynchronous and uses a callback object to signal completion. Implement onCompletion(metadata, exception) to handle success or failure, and use metadata for topic, partition, offset, timestamp.

Implement a Kafka producer callback in Java with a myCallback class that overrides onCompletion to report metadata, offset and topic, and configure batch size, linger.ms, and buffer.memory for topic mytopic2.

Explore producer callbacks in Kafka by sending 20 messages with keys and values to topic mytopic2, observe callback invocation in parallel threads, and inspect producer properties and partition distribution.

Simulate a synchronous Kafka producer by omitting a callback and calling get on the future from send; the program waits for the broker and handles exceptions.

Demonstrate near synchronous kafka producer behavior in java by using the get method on producer.send and print record metadata as the broker acknowledges 20 messages to topic mytopic2.

Produce messages in Apache Kafka by configuring serializers for key and value types, creating a producer with the correct types, sending producer records to a topic, and closing the producer.

Demonstrate handling integer keys and double values by configuring a producer with integer and double serializers, sending twenty messages to topic mytopic3 for verification with a consumer.

Demonstrate configuring the producer and console consumer to handle integer keys and double values by setting key.deserializer and value.deserializer to IntegerDeserializer and DoubleDeserializer, ensuring correct serialization and deserialization.

Learn how Kafka assigns messages to topic partitions with the default partitioner or a custom partitioner, and implement a custom partitioner by returning a 0 to N-1 partition number.

Explore implementing a custom partitioner in Kafka by creating key_partitioner, using key modulo partition count, and wiring it into the producer with configured serializers and properties.

Create and run a Kafka producer in Eclipse, sending messages to a topic and exploring producer configurations, callbacks, and near-synchronous delivery; master serializers, deserializers, and a custom partitioner.

Explore Kafka consumer APIs to read and process messages, experiment with keys, values, and deserializers, and implement parallel consumption, consumer groups, and broadcast or publish-subscribe patterns through hands-on coding.

Kafka consumer reads messages from a topic as a client program, deserializing data from partitions managed by the broker, with multiple consumers in a consumer group reading in parallel.

Explore how Kafka partitions preserve message order and how offsets track each consumer's position. See that messages stay on the broker and only a consumer's offset advances as it reads.

Configure a Kafka consumer with java properties, set bootstrap.servers, groupid, deserializers, and auto commit. Then subscribe, poll for records, process them, handle errors, and commit offsets for recovery.

Demonstrates a simple java consumer that connects to a Kafka broker, subscribes to a topic, polls for messages, prints offsets, keys, and values, and commits offsets.

Practice activity demonstrates running a simple consumer program and shows how producer batching and linger.ms shape message delivery and offsets.

Explore the consumer program from top to bottom, configuring properties, poll timeouts, and client libraries; set topic mytopic1 and group myGroup1 to read messages.

Explore Kafka consumer configuration properties such as bootstrap.servers, key/value deserializers, and group.id, and learn how heartbeat intervals and session timeouts govern liveness and offset commits.

Understand consumer properties like max.poll.records and auto.commit with defaults, and use the TopicPartition class to represent topic and partition, enabling multi-topic and multi-partition consumption.

Learn to configure a kafka consumer with localhost:9092, group id, string deserializers, and manual commit. Then poll for messages, process records, and commit offsets for fault tolerance.

Configure the Kafka consumer using the properties object to set key.deserializer and value.deserializer so you can read messages produced with int keys and double values.

Demonstrates building an Apache Kafka consumer that reads integer keys and double values from topic mytopic3, using integer deserializers, auto commit, and printing full message details.

Demonstrate consuming messages with integer keys and double values using the consumer API, while producing them to a topic with the producer API.

Explore how Kafka assigns each partition to a single consumer to preserve order. Learn how rebalancing occurs with group changes and how separate groups enable broadcasting.

Demonstrates how consumer groups and partitions interact in Kafka by running multiple consumers with the same group id and a producer to show message distribution across a three-partitions topic.

Run multiple consumer instances with the same and different group ids to observe how messages are distributed across partitions, while a producer and partitioner populate partitions 0, 1, and 2.

Demonstrates how consumer groups and partitions behave in Kafka, showing partition assignment and rebalancing as consumers join or fail, and illustrating fault tolerance in a multi-group setup.

Learn to reposition a Kafka consumer with seek, seekToBeginning, and seekToEnd to reread messages from a chosen offset, ensuring topic partitions are assigned.

Demonstrate changing consumer offsets in a kafka_conSeek demo, using a TopicPartition for topic mytopic1 partition 0, subscribing, and seeking to a specified offset or to beginning or end.

Demonstrate adjusting a Kafka consumer offset by inserting a dummy poll before seek, resolving illegal state errors from missing assignment, and exploring offset progression from 10 to 59.

Create a consumer using the consumer API in Eclipse, subscribe to topics, and deserialize messages with key/value types. Explore consumer groups, parallel consumers, assignment, rebalancing, offset seek, and producers.

Develop as an Apache Kafka developer by building producer and consumer apps, creating topics, and processing messages with keys and values via Kafka APIs and consumer groups.