Apache Kafka for developers

Explore Kafka concepts, use cases across industries, the core data model of messages and topics, and the producers, consumers, and brokers in distributed, fault-tolerant Kafka architecture, with hands-on topic creation.

Explore how Kafka ingests streams from thousands of sources for real-time analytics. See how data is stored in Hadoop or Cassandra, processed by Spark, and visualized in Tableau.

Learn how Kafka treats every data item as a message or record with a key, value, and timestamp, and how messages are organized into topics like logrecords and salesdata.

Learn how producers and consumers interact with Kafka topics, using console producer and console consumer for testing, and Java APIs to produce and consume messages.

Kafka brokers are processes in a multi-node cluster that receive messages from producers, store them in topics, and deliver messages to consumers on request while tracking each consumer's progress.

Serialization and deserialization, or ser-de, convert program objects to and from a stream of bytes, with Kafka serializers for common types like string, integer, long, double, and bytes.

create topics with the kafka-topics cli by specifying a name, partitions, and replication factor, connecting to zookeeper at localhost:2181 and ensuring replication factor does not exceed broker count.

Master the kafka-topics command to list topics and create new topics with partitions and replication factors, and learn how replication factor must not exceed available brokers.

Learn to send and read messages in Kafka cluster without coding, using kafka-console-producer and kafka-console-consumer. Connect to localhost:9092 with bootstrap-server and topic newtopic1 to see messages appear in real time.

Demonstrate creating and reading messages with keys and values using the Kafka console producer and consumer, configuring key separators and print key options on topic newtopic1.

Choose suitable hardware and operating system for Kafka installation, select a Kafka version, and install the software; start Kafka and Zookeeper with minimal configuration while reviewing essential development options.

Plan hardware and operating system choices for Kafka deployments, detailing development and production memory, CPU, storage requirements, and zookeeper fault tolerance.

Install kafka on linux from a tarball into /usr/local/kafka, set JAVA_HOME and PATH, and start zookeeper and kafka in the background with nohup; stop using the stop scripts.

Install Apache Kafka 2.2.0 on Ubuntu 16, extract the tarball to /usr/local/kafka, configure the path, start zookeeper and Kafka servers with nohup, manage topics, and stop the services.

Explore Kafka broker configurations from server.properties, including zookeeper.connect, broker.id, log.dirs, log.retention.hours, and listeners such as plaintext on port 9092.

Fix startup failures on Windows by clearing all log files in Kafka-logs, then restart the server; if it aborts again, reinstall Kafka, not for production.

Install and start ZooKeeper and Kafka servers across Windows, open source, and Confluent options, and compare development and production configurations. Review the ZooKeeper bundle and broker configuration options.

Set up Eclipse for Kafka development by configuring the workspace, adding the JRE system library (JDK 1.8), adjusting compiler settings to 1.7, and building the Kafka_prodSimple project.

Open and inspect the pom.xml in kafka_prodSimple to manage maven dependencies, including kafka version 2.1.0, kafka clients with provided scope, log4j libraries, and jdk.tools path, enabling compile-time Kafka programs.

Master Kafka producers by building and running producer programs, using callbacks and synchronous processing, exploring keys, values, serializers, and custom partitioning logic.

Add messages to a Kafka topic by sending them to the broker. Target one or more partitions, serialize data before sending, and preserve order as received.

Use the Kafka producer API to publish messages to a topic by configuring a Java properties object, setting broker address, batch size, serializers, then create and close the producer.

Explore configuring a Kafka producer in eclipse by adjusting bootstrap.servers, acks, batch.size, linger.ms, and buffer.memory; review serializers and producer records for multiple topics, and use asynchronous sends with futures.

Implement a Kafka producer callback in Java with a myCallback class that overrides onCompletion to report metadata, offset and topic, and configure batch size, linger.ms, and buffer.memory for topic mytopic2.

Learn how Kafka assigns messages to topic partitions with the default partitioner or a custom partitioner, and implement a custom partitioner by returning a 0 to N-1 partition number.

Explore Kafka consumer APIs to read and process messages, experiment with keys, values, and deserializers, and implement parallel consumption, consumer groups, and broadcast or publish-subscribe patterns through hands-on coding.

Configure a Kafka consumer with java properties, set bootstrap.servers, groupid, deserializers, and auto commit. Then subscribe, poll for records, process them, handle errors, and commit offsets for recovery.

Practice activity demonstrates running a simple consumer program and shows how producer batching and linger.ms shape message delivery and offsets.

Understand consumer properties like max.poll.records and auto.commit with defaults, and use the TopicPartition class to represent topic and partition, enabling multi-topic and multi-partition consumption.

Learn to configure a kafka consumer with localhost:9092, group id, string deserializers, and manual commit. Then poll for messages, process records, and commit offsets for fault tolerance.

Demonstrate consuming messages with integer keys and double values using the consumer API, while producing them to a topic with the producer API.

Demonstrates how consumer groups and partitions interact in Kafka by running multiple consumers with the same group id and a producer to show message distribution across a three-partitions topic.

Run multiple consumer instances with the same and different group ids to observe how messages are distributed across partitions, while a producer and partitioner populate partitions 0, 1, and 2.

Demonstrates how consumer groups and partitions behave in Kafka, showing partition assignment and rebalancing as consumers join or fail, and illustrating fault tolerance in a multi-group setup.

Learn to reposition a Kafka consumer with seek, seekToBeginning, and seekToEnd to reread messages from a chosen offset, ensuring topic partitions are assigned.

Demonstrate changing consumer offsets in a kafka_conSeek demo, using a TopicPartition for topic mytopic1 partition 0, subscribing, and seeking to a specified offset or to beginning or end.

Demonstrate adjusting a Kafka consumer offset by inserting a dummy poll before seek, resolving illegal state errors from missing assignment, and exploring offset progression from 10 to 59.