Apache Kafka A-Z with Hands on Learning

Understand data evolution over time and gain a big data overview, analytics, and messaging system context. Explore Kafka components and architecture, cluster types, and real-world industry use cases.

Explore how digital data evolves from binary storage to structured, semi-structured, and unstructured forms, with examples from emails, documents, and media, and examine rapid unstructured data growth driving big data.

Explore big data as a massive, diverse data landscape that challenges traditional processing with velocity, volume, variety, value, and veracity, using social media and sensor data scales.

Explore big data analytics to uncover hidden patterns, unknown correlations, and market trends that help organizations make informed business decisions using cost-saving technologies and in-memory analytics.

Explore the basics of a publish-subscribe messaging system, its challenges with many publishers and subscribers, and how Kafka acts as a distributed broker to decouple data pipelines.

Discover Kafka as a distributed publish-subscribe system with a storage layer, enabling real-time metrics and user activity data at scale, with on-disk persistence, no data loss, compression, and security.

Discover Kafka use cases such as messaging, metrics and logging, and stream processing. Examine user activity tracking, log aggregation, event sourcing, and real-time monitoring with commit log and long aggregation.

Understand Kafka terminologies: a message is a unit of data with a key value pair, a badge groups messages, topics partition into partitions, producers publish to topics, consumers subscribe.

Explore Kafka components including topics, partitions, offsets, producers, and consumers, and learn how batching, compression, and round-robin distribution affect latency, throughput, and data retention across a distributed cluster.

Explore the Kafka architecture, where producers publish messages to topics and consumers read subscribed topics, in a multi-broker cluster distributing partitions; topic a has two and topic b has three.

Explore Kafka cluster types, including single-node single-broker, single-node multi-broker, and multi-node multi-broker configurations, where a broker acts as controller and Kafka and Zookeeper run as multiple instances.

Explore real-world Kafka use cases, from tracking page views, click data, and user profiles to Twitter's migration for lower latency and cost savings, plus global log collection with offline resilience.

learn to install zookeeper on Windows, edit zookeeper properties and server properties files, download and extract the zookeeper package, and start zookeeper using batch files.

Install Kafka on a Windows system, start Zookeeper on the same machine, configure server properties and logs path, then run Kafka server from bin Windows folder and create topics.

Demonstrates how to set up a single node kafka cluster by downloading kafka and zookeeper, configuring server and zookeeper properties, and starting zookeeper then the kafka broker.

Configure a multi-node Kafka cluster by editing broker IDs and listeners, configure Zookeeper on port 2181, start Zookeeper, then launch multiple Kafka brokers using server properties and Zookeeper connection strings.

Understand data evolution over time, big data analytics, and messaging systems, with Kafka overview, COFCO components and architecture, cluster types, and Kafka industry use cases.

Configure COFCO producers and create a COFCO producer to send messages to Kafka, send messages synchronously and asynchronously, serialize messages using Apache Avro, and handle topic partitions.

Explore the Kafka producer and its architecture, including why to write messages for activity, auditing, and asynchronous communication, plus how records, serializers, partitions, and retries affect latency and throughput.

Explore Kafka producer configuration essentials, including bootstrap servers and mandatory serializers for keys and values, and learn how to implement custom serializers.

Explore Kafka producer optional configurations, including memory, batch sizes, retries, timeouts, and acks settings (0, 1, all) to balance throughput and reliability across replicas.

Configure Kafka producer settings to balance latency and throughput, controlling buffer memory, compression type (snappy, Z4), retries, batch size and linger, client id, max in-flight requests, and timeouts.

Create a Kafka producer using bootstrap servers, string serializer for key and value, and a producer record to publish messages to a topic, then verify with a console consumer.

Explore how the Kafka producer sends messages to the broker via fire-and-forget, synchronous, and asynchronous sends, with topic, key, and value serialization, and error handling.

Explore why serializers are required and how producer and consumer data become byte arrays in kafka, noting that custom serializers incur overhead and that key and value types must match.

Demonstrates creating a COFCO serializer by extending the serializer interface and implementing configure, serialize, and close; uses it in coproducer to send a student object as bytes to Kafka.

Learn how Avro enables schema evolution and compatibility in Kafka by standardizing serialization between producers and consumers, using a schema registry to store schemas and enable binary, compressed records.

Demonstrates serializing an Avro schema (student with name and age) to Kafka as a byte array, sending to demo topic three, and verifying receipt with a console consumer.

Learn how Kafka topics contain multiple partitions and how keys map to partitions, ensuring the same key goes to the same partition, with round-robin fallback when keys are absent.

Learn how to configure a custom partition in Kafka, distribute records by hash of the key, create a four-partition topic, and verify data delivery with producer and consumer.

Understand the Kafka producer architecture, explore different producer configuration options, learn methods of sending messages, and master serializers including Avro and custom serialization, and partitions.

Explore the Kafka consumer basics by examining consumer groups and partition rebalance, understand their functioning, and configure consumers with offset rebalance listeners to consume records at specific offsets.

Understand how a Kafka consumer subscribes to topics or assigns partitions, and how consumer groups rebalance load. Learn how standalone consumers differ, and how messages are validated for processing.

Explore how Kafka consumer groups distribute topic partitions among members, enable scalable reading, and coordinate heartbeats with a group coordinator to rebalance on member changes.

Explore partition rebalance dynamics and how to create a COFCO consumer with bootstrap servers, key/value deserializers, and a group id, then subscribe to topics.

Poll loop drives the api by polling data from partitions via the poll method with a timeout, handling coordination and rebalances while returning topic, partition, and record data.

Configure Kafka consumers by tuning fetch min bytes, fetch max wait, and max partition fetch bytes, while managing session timeout to balance failure detection and rebalance.

Control Kafka consumer offset behavior and reset options (latest or earliest), auto commit settings, then explore partition assignment strategies like range and round robin with client id and max.poll.records tuning.

Create a Kafka consumer, configure bootstrap servers, key and value serializers, and a consumer group, then consume and print records from a demo topic two.

Explore how commits and offsets track consumer progress in Kafka, covering per-partition offsets, consumer lag, rebalance effects, and how mismatches can cause duplicates or missed messages.

Explore automatic offset committing in the Kafka consumer, driven by the poll loop, with a default five-second interval, balancing duplicates risk after rebalances.

Learn manual offset committing with commitSync to commit the latest offset from poll after processing, preventing misses and duplicates during rebalances, while trading throughput for reliability.

Use asynchronous commit to improve throughput by not waiting for broker responses, but expect duplicates during rebalance and no retries; provide a commit async callback for logging errors or metrics.

Combine asynchronous and synchronous commits to manage Kafka consumer offsets, applying async after every poll and sync during rebalance or shutdown to ensure retriable and safe commits.

Learn how to commit specified offsets mid batch using commit sync or async with a map of partitions and offsets to handle rebalances.

Learn how rebalance listeners manage cleanup and offset commits during partition rebalance, with on partition revoked and on partition assigned callbacks to maintain data integrity.

Learn to consume records from offsets in Apache Kafka using seek to beginning, seek to end, and seek to offset, and exit the poll loop with wake up and close.

Kafka producers serialize objects to byte arrays and consumers deserialize them; ensure producers and consumers share compatible serializers like Avro or Thrift, avoiding custom deserializers due to fragility.

Implement a custom student serializer and configure a Kafka consumer to use it, then produce a student record and verify the consumer receives name and age.

Explore the overview of consumer groups and partitions, understand rebalance and its functioning, and learn configuring consumers, commit and offsets, rebalance listeners, and consuming records with specific offsets.

Explore COFCO internals and performance tuning by detailing cluster and controller replication, request processing, storage, partition allocation, index file, and compaction for reliability, plus configuration and validation.

Explore how Kafka uses ZooKeeper to manage cluster membership, register brokers with ephemeral nodes, and elect a controller that handles partition leaders, with watches and reselection.

Kafka replication ensures durability and high availability by maintaining multiple topic-partition replicas across brokers, with leaders handling requests and followers staying in sync.

Identify the preferred leader among replicas to balance partition load across brokers and trigger elections when the preferred replica is in sync, with clients using metadata requests to locate leaders.

Explore the types of requests in Kafka, including produce and fetch requests, how brokers validate data, manage offsets, and deliver messages to the leader and in-sync replicas.

Discover how kafka uses partitions, replication, and rack awareness to provide reliable availability across brokers, then manage data with retention and segments, including active segments and sizing effects.

Describe how Kafka stores messages in segment files with on-disk format, index and timestamp indexes, and how compaction keeps the latest value per key, plus commit options for multi-partition consumers.

Explore how Kafka ensures reliability across producers, consumers, and brokers with partition order guarantees and acks. Review replication factor, in-sync replicas, and retries and commit configurations for dependable delivery.

Configure replication factor and min in-sync replicas at broker and topic levels to balance reliability and costs. Learn about uncleaned leader election and not enough replicas exception.

Explore how proper producer configuration, acks, retries, and error handling ensure reliability in Kafka, including handling leader crashes, replicas, and strategies to detect and avoid duplicates.

Learn how to configure a reliable Kafka consumer, manage offsets and group IDs, and use verifiable producers and consumers to validate and monitor consumption, including leader elections and rolling restarts.

Gain understanding of Kafka internals, overview, cluster and controller replication, request processing storage, partitions, location index, fine compaction, reliability, configuration, and validation.

Explore Kafka cluster architectures and administration, learning to configure and manage cluster mirroring, mirror maker, dynamic configuration changes, and the console producer and consumer tools.

Understand cross-cluster mirroring in Kafka with Mirror Maker to copy data between regional and central clusters, covering use cases, data redundancy, cloud migration, and best practices.

Explore multi cluster architectures for Kafka, including hub and spoke, active-active, active-standby, and stretch clusters. Learn how local clusters mirror data to a cluster and evaluate access and fault tolerance.

Explore active-standby and stretch cluster architectures in Apache Kafka to achieve data center redundancy, synchronous replication, and efficient resource use across three data centers.

Understand MirrorMaker’s cross-cluster data copy, its five configuration parameters, and production deployment, including multi-consumer single-producer setup, offset commits, and 60-second replication windows.

Deploy MirrorMaker in production with Docker or automation tools and scale across instances; monitor lag and offsets, and tune producer and consumer settings for throughput and latency.

Set up and run Kafka mirror maker to replicate data between source and destination clusters, configuring consumer and producer properties and validating replication with a live message.

Master topic operations in Kafka using the command line, including creating, listing, describing, and deleting topics with partitions and replication. Learn constraints like partition rebalancing and irreversible deletes.

Explore Apache Kafka consumer group operations, including list group, describe group, delete group, and offset management, using ZooKeeper for old clients or bootstrap server for new clients.

Master dynamic configuration changes by overriding cluster and topic defaults at runtime with ad config parameters, and manage clean up, retention, and flush-to-disk settings.

Explore partition management in Apache Kafka by understanding leader and in-sync replicas and performing preferred replica elections. Learn to move partitions with the kafka-reassign-partitions tool using JSON configurations and verification.

Learn to use Kafka console producer tool to publish messages to topic, with line per message, tab-separated key and value, and configure producer properties via file or command line interface.

Master the Kafka console consumer to manually consume messages from one or more topics in a cluster, with options for old and new modes, filtering, and formatters.

Gain understanding of cluster mirroring, multi cluster architecture, mirror maker configuration deployment and tuning, administrating Kafka dynamic configuration changes, and using console producer and console consumer tools.

Explore COFCO monitoring and Skybound Registry concepts, mastering the Schema Registry architecture, components, metrics, and practical use of COFCO Schema Registry.

Learn the most critical metrics to monitor in Kafka and how to respond to them, with a focus on debugging metrics and overall performance.

Explore server and IT infrastructure monitoring with tools like New Relic to detect issues before they affect availability and performance, and master application metrics, thresholds, and alerts to maintain levels.

Monitor Kafka performance by tracking producer and consumer behavior and lag with open-source and paid tools. Measure metrics such as batch size, latency, memory, and offsets via JMX.

Monitor under replicated partitions to keep this metric at zero and alert when it rises. Distinguish broker-level from cluster-level issues, and diagnose with hardware, system, and configuration checks.

Examine Kafka broker metrics such as the active controller count, partition leadership, request metrics, and topic bytes and rate measurements to monitor cluster health and performance.

Explore topic and partition specific metrics in Kafka, including bites in rate, bites out rate, failed fetch rate, and partition size and offsets.

Learn how to monitor Kafka health through targeted logging, enable loggers at the appropriate levels, and track broker status, topic creation and modification, and producer and consumer activities.

Explains producer and consumer metrics in Kafka, covering record error and retry rates, request latency, bytes per second, and per broker and topic metrics with quotas.

Monitor consumer lag by tracking partition states and offsets with external tools, using Burrow or Kafka Offset Monitor to track topics, consumer groups, and lag.

Explore a monitoring dashboard to view Kafka clusters and producer-consumer activity with the Kafka Offset Monitor. Compare topic lag across Kafka Monitor, Boru, Confluent Control Center, and Prometheus-Grafana for visibility.

Learn how to set up open source Kafka Offset Monitor to track consumer offsets, lag, and activity, including installing the jar, configuring offset storage, and verifying via a browser.

Explore Kafka Schema Registry as a middleware that handles diverse source data and evolving Avro schemas, enabling backward and forward compatibility while managing serialization between JSON and binary byte arrays.

Explore Kafka architecture and schema registry concepts, including subjects, versions, and IDs, and learn how producers and consumers use serializers with ZooKeeper and brokers.

Explore how Kafka schema registry stores a version history of Avro schemas, manages compatibility and evolution across producers and consumers, and enables concurrent, low-coordination schema upgrades for Kafka ecosystems.

Explore how the Apache Kafka schema registry operates using Java to develop, register, and retrieve Avro schemas with version IDs and compatibility checks.

Learn to set up the Kafka schema registry with Kafka and Zookeeper, configure compatibility for evolving schemas, and run multiple registry instances.

Gain practical insight into monitoring metrics, the architecture and components of Kafka, and how the Kafka schema registry works.

Explore Kafka's dreams and its architecture components and topology, learn about processors, stream concepts, and Kafka connectors and their configurations.

Discover Kafka streams, immutable key–value records, and real-time processing in motion, with end-to-end exactly-once delivery, replay at requested positions, and practical use cases.

Explore Kafka streams, a client library for building real-time applications with data stored in Kafka clusters, delivering fault-tolerant, elastic, and scalable stateful processing with low latency.

Kafka streams architecture, including producers, consumers, partitions, and streaming topology, with local state stores and fault tolerance for scalable, stateful stream processing.

Discover Kafka streams components, including stream partitions, stream tasks, processor topology, and state stores. Learn how threads and memory support parallelism, with fixed partition-to-task assignments and record buffers.

Explore Kafka streams architecture with per-partition tasks, sub-topologies, and parallel processing. Learn about threading, local state stores, stateless versus stateful processing, and fault-tolerant interactive queries.

Understand Kafka stream architecture via the record buffer, a per-thread record cache that speeds reads from the state store, batches writes, and tracks changelog activity.

Master memory management in Kafka streams by configuring ram, caches, and buffering to optimize state stores, key table updates, and real-time streaming pipelines.

Explore Kafka streams DSL and the stream-table duality, treating streams as tables and vice versa for real-time data processing. Build key streams, apply lowercase transformations, and publish results to topics.

Explore KStream operations like filter, flat map values, branch, and group by with code examples, applying transformations on source streams to destination topics.

Explore how a key table reads an input topic into a key table and treats records with the same key as inserts or updates in a changelog stream.

Learn k-table operations in Apache Kafka, including inner and outer joins (left, right, full), to update and enrich streams like customer info and orders with destination topic output.

Explore aggregation in Kavkaz Streams DSL, transforming multiple records into one per key via group by, with windowing that defines per-window stateful results. Examine tumbling, hopping, and session windows.

Define processor topology as a graph of nodes connected by streams and stores, and learn source, issuing, and sync processors, plus how to implement topology with APIs or DSL.

Explore the stream processor as a node in a topology that pulls streams from Kafka, applies transformations, and forwards to downstream processors, using stateless and stateful processing with state stores.

Create a Kafka streams topology with a source processor reading from a credit card topic, a masking stream processor, a destination topic, and a state store.

Explore Kafka connectors, including source and sync connectors, to move data between external sources and Kafka topics, with standalone and distributed modes, and management via rest APIs.

Configure standalone and sink connectors in Apache Kafka by setting bootstrap servers, key/value converters, and optional schemas, plus offset storage, to enable reliable data pipelines.

Run Kafka Connect using standalone mode with source and sink connector properties, streaming JSON records from logs.txt into a connect topic via the console producer and consumer.

Explore Apache Kafka connectors in distributed mode, configuring multiple worker nodes, broker addresses, and converters for keys and values, with a rest interface, storage topics, and high-availability setup.

Learn to use the http rest interface of Koffler Connect in distributed mode, with a json rest api to list, create, and fetch connectors and configs, and inspect worker details.

Explore Apache Kafka connectors that stream data between Apache Kafka and external systems, using Kafka Connect distributed or standalone to move data into and out of Kafka.

Demonstrates creating a COFCO/Kavkaz stream pipeline with a stream builder, converting messages to lowercase, and routing to a destination topic, plus word count.

Gain a solid understanding of Kafka stream architecture, stream concepts, and how processors and topology work. Explore Kafka connectors and configure connector configurations.

Explore Storm and its features, architectures, and components. Learn how COFCO fits in with Storm and how to integrate Storm with Kafka.

Explore Apache Storm's real-time streaming capabilities, a distributed framework designed for high ingestion rates. It is fault-tolerant, scalable, and uses Zookeeper to manage Nimbus and Supervisor in a master-slave architecture.

Learn real-time analytics for stream processing with Kafka as the backbone, capturing on-the-fly data and applying immediate processing with Spark or Storm, streaming results to dashboards.

Understand the storm architecture with a single Nimbus master, multiple supervisors, and Zookeeper coordinating a stateless, distributed cluster with failover to keep processing running.

Explore Apache Storm components, including Nimbus, supervisors, and workers, and how Zookeeper coordinates heartbeats, failover, and task distribution for storm topologies.

Describe how Apache Storm topology visualizes computation as a directed graph of spouts and bolts that process streams in real time.

Explore Apache Storm topology concepts, including spouts, bolts, and tuples, and design a Java-based topology with stream grouping, parallelism, and distributed execution.

Integrate Storm with Kafka using the Kafka spout, a consumer that pulls streams from topic partitions for real-time analytics while configuring brokers, ZooKeeper, topic, group id, and serializers.

Develop a word count app by streaming sentences from Kafka and Storm, using a Kafka spout, split line bolt, and count bolt to tally words in a topology.

Learn to integrate Kafka with Storm by building a topology of spouts and bolts that splits lines into words and counts frequencies for real-time streaming analytics.

Explore Storm's architecture and components, map its topology, and implement Kafka in Storm to integrate Apache, Storm, and Kafka.

Explore integration of Spark with Flume and Kafka, covering Spark components, basics of data sets, transformations, architecture, configuration and connectors, and Kafka to FGF configuration.

Explore how Kafka and Spark integrate to build end-to-end real-time applications, highlighting Spark's in-memory, batch and stream processing, graph theory API graphics, and polyglot language support.

Explore the Spark components stack from Spark Core with RDD-based transformations to Spark Streaming, MLlib, and GraphX, enabling in-memory processing, real-time analytics, and graph computations.

Explore how the resilient distributed dataset underpins Spark applications with partitioned in-memory processing, transformations and actions, and fault-tolerant execution via Spark context and executors.

Explore spark transformations and actions, focusing on map, flatMap, and filter, with lazy evaluation and lambda expressions guiding functional operations on RDDs.

Explore rdd transformation operations such as inner joins and left, right, and full outer joins on key-value pairs, plus distinct and actions like first and take to trigger execution.

Discover how RDD actions like reduce and reduce by key operate on key-value pairs across partitions. Count triggers execution and flatMap splits sentences into words to count totals.

Understand data sets and the Spark session, including the data set API in Spark 2.0, the shift from data frames to typed datasets, and megastore access through Hive and Impala.

Create and access a Spark session using the builder pattern, switch from Spark context to Spark session, and read data with schemas to work with data frames and datasets.

Learn how Spark Streaming ingests data from sources like Kafka and Kinesis, applies complex logic in real time, and stores results in DFS or cloud storage with dashboards.

Integrate spark with kafka to build a spark kafka application using a kafka configuration and a spark streaming context, processing dstreams from topics with a time window.

Learn to integrate Kafka with Spark streaming to build real-time applications by consuming from Kafka and counting word frequencies in the last 60 seconds, with configurable duration and direct streams.

Explore Apache Flume as a simple, scalable stream capture tool for Hadoop, detailing its events, sources, sinks, and channels, and learn to configure it with a config file.

Explore how Flume connectors read data from S3, convert files into streams, and push them to Kafka, enabling seamless data transfer between Hadoop and streaming tools.

configure flume to ingest kafka streams and write to hdfs via a kafka source and memory channel, storing time-stamped files from the transaction data topic with 1000 events per file.

Configure an Apache Flume agent to read from a Kafka source and write to HDFS, using a dockerized Hadoop environment and a Kafka topic for end-to-end data flow.

Summarize the key takeaways by understanding spark and its components, data sets, transformations, actions, streams, architecture, configuration, connectors, and integration with Kafka and Flume for HDFC configuration.