Apache kafka - A Complete Hands-on Kafka Developer's Guide
4.3 (26 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
2,622 students enrolled

Apache kafka - A Complete Hands-on Kafka Developer's Guide

Learn Kafka Ecosystem, Architecture, Core & Master Concepts, Operations, Console-producer, Twitter Producer & More
4.3 (26 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
2,622 students enrolled
Created by Apex Education
Last updated 10/2019
English
English [Auto]
Current price: $13.99 Original price: $19.99 Discount: 30% off
23 hours left at this price!
30-Day Money-Back Guarantee
This course includes
  • 8 hours on-demand video
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • Understand Apache Kafka Ecosystem, Architecture, Core Concepts and Operations
  • Master Concepts such as Topics, Partitions, Brokers, Producers, Consumers
  • Start a personal Kafka development environment
  • Learn major CLIs: kafka-topics, kafka-console-producer, kafka-console-consumer, kafka-consumer-groups, kafka-configs
  • Create your Producers and Consumers in Java to interact with Kafka
  • Program a Real World Twitter Producer &ElasticSearch Consumer
Requirements
  • A recent Windows / Mac / Linux machine with minimum 4GB of RAM, 5 GB of disk space
  • Some understanding of Java Programming
  • Good to have knowledge about Linux command line
  • Desire to learn something awesome and new!
Description

Apache Kafka has become the leading distributed data streaming enterprise big data technology. Kafka is used in production by over 33% of the Fortune 500 companies such as Netflix, Airbnb, Uber, Walmart, and LinkedIn.

To learn Kafka easily, step-by-step, you have come to the right place! No prior Kafka knowledge is required.

If you look at the documentation, you can see that Apache Kafka is not easy to learn...

Thanks to my several years of experience in Kafka and Big Data, I wanted to make learning Kafka accessible to everyone.

We'll take a step-by-step approach to learn all the fundamentals of Apache Kafka.

At the end of this course, you'll be productive and you'll know the following:

  • The Apache Kafka Ecosystem Architecture

  • The Kafka Core Concepts: Topics, Partitions, Brokers, Replicas, Producers, Consumers, and more!

  • Launch your own Kafka cluster in no time using native Kafka binaries – Windows / MacOS X / Linux

  • Learn and Practice using the Kafka Command Line Interface (CLI)

  • Code Producer and Consumers using the Java API

  • Real-world project using Twitter as a source of data for a producer and ElasticSearch as a sink for our consumer


Note: The hands-on section is based on Java, which is the native Kafka programming language. But, good news! Your learning in Java will be completely applicable to other programming languages, such as Python, C#, Node.js or Scala, and Big Data frameworks such as Spark, NiFi or Akka.

Who this course is for:
  • Developers who want to learn the Apache Kafka Fundamentals, start a cluster and write their first application
  • Architects who want to understand how Apache Kafka fits into their solution architecture
  • Anyone looking to learn the full theory of how Apache Kafka works as a distributed system
Course content
Expand all 65 lectures 07:59:35
+ Module 1: Introduction to Apache kafka
6 lectures 30:42

This lecture provides an overview of the course and the course objectives.

Preview 01:39

This video describes the prerequisites in order to be able to setup your own development

Preview 01:33

This lecture explains how kafka works, how it will help you and some use cases of Kafka

Preview 07:09

In this video we talk about the kafka core and we will explain the kafka extended API, the confluent components schema registry and REST proxy and finally we list the administration and monitoring tools

4- Kafka and the kafka ecosystem high level overview
07:38

Here we talk about the apache Kafka architecture for batch and real time processing, and we will an overview about the highly  available multi-datacenter streaming, using confluent to manage your transition to the cloud or back and the automatic data balancing

5- Kafka architecture within the enterprise
09:08

this video contains a short review of topics from the "Introduction to Apache Kafka" lecture

6- Review questions
03:35
+ Module 2: Apache Kafka Core Concepts
10 lectures 01:02:06

This lecture explain the first important components of Apache Kafka, we will describe the topics and partitions by using examples

1- Topics and partitions
04:53

This lecture explains another important component of Apache Kafka, we will describe the brokers, data replication and the concept of leader by using examples. finally, we will talk about “how to create a kafka Topis” by giving command lines

2- Brokers and data replication
10:07

This lecture represents the working method of producers, we describe how producers write data to topics, we will talk about the acknowledgment of data writes and finally the message keys of the producer.

3- Kafka Producers
08:25

This lecture represents the principle of consumers and consumer groups and finally we will describe data availability of consumers 

4- Kafka Consumers and Consumer Groups
06:58

This video represents the advantage of Consumer offsets and how consumer can be able to read back the data due to consumer offsets, finally we will talk about the method that consumers choose when to commit offsets

5- Consumer Offsets & Delivery Semantics
07:23

in this video we will talk about Kafka broker discovery and the concept of bootstrap servers

6- Kafka Broker Discovery
04:04

in this video we will talk about Kafka broker discovery and the concept of bootstrap servers

7- Zookeeper
07:43

This lecture represents the Kafka guarantees

8- Kafka Guarantees
02:00

this video shows you a diagram which sums up all what you learned until now

9- Theory Roundup
03:16

this video contains a short review of topics from the "Apache Kafka core concepts" lecture

10- review questions
07:17
+ Module 3:: Starting Kafka
3 lectures 26:20

This lecture explains how to download and install Kafka on centos 7.

1_Download_install_Kafka_Linux
09:12

this video is following the previous video, it will explain how to start Kafka service on centos 7.

02_Start_Kafka_Zookkeeper_Linux
11:07

This lecture explains how to download, install and start Kafka on windows 10.

03_Download_Start_Kafka_Zookeeper_Windows_10
06:01
+ Module 4: Command Line Interface (CLI)
10 lectures 01:24:58

This video discusses about the CLI introduction

1_CLI Introduction
00:37

this lecture explain how to create a topic on CLI and also some other important commands

2_Kafka Topics CLI
12:17

this video explain how to manipulate producer on CLI, we will the most important command about producers

3_Kafka Console Producer CLI 1
07:53

this lecture explain how to read messages sent by producers and other options about consumers

3_Kafka Console Producer CLI 2
08:41

this video shows how to use kafka consumer groups command to manipulate our consumer groups

4_Kafka Console Consumer CLI
08:18

his lectures represents the important of consumers groups and how to use them

5_Kafka Consumers in Group 1
06:13
5_Kafka Consumers in Group 2
09:55

this video shows how to use kafka consumer groups command to manipulate our consumer groups

6_Kafka Consumer Groups CLI
11:45

In this video we will learn how to reset offsets for the topics.

7_Resetting Offsets
10:59

this lecture explain an important UI tool of apache Kafka it’s called kafka tool, it’s like the command line but with UI.

8_Kafka Tools UI
08:20
+ Module 5 : Kafka Java Programming
11 lectures 01:47:11

Introduction to kalfka Java Programming

1_Intro to Kafka Programming
00:38

this video represents how to create a new project in IntelliJ IDE and do a “Hello world”.

2_Creating Kafka Project
13:37

This video represents a hands-on to create a producer in java and check the message send to the consumer.

3_Java Producer 1
09:00
3_Java Producer 2
10:00

this video explain the callbacks for producer in java

4_Java Producer Callbacks
12:12

this video explains how to order messages by using keys for producers

5_Java Producer with Keys
13:30

this video explain how to use consumers with java

6_Java Consumer_1
12:46
6_Java Consumer_2
12:41

this lecture represents the rebalancing feature of consumer groups with java

7_Java Consumer inside Consumer Group
11:26

this video presents the seek and assign in a java class

8_Java Consumer Seek and Assign
09:18

this lecture represents the client Bi-Directional Compatibility

9_Client Bi-Directional Compatibility
02:03
+ Module 6 :Kafka Twitter Producer & Advanced Configurations
16 lectures 01:43:26

This video describes about the Realworld project overview

1_Real World Project Overview
01:08

this represents an overview of producer and advanced configurations

2_Producer and Advanced Configurations Overview
00:48

this video represents how to create an application in twitter developer account

3_Twitter Setup
12:33

this video it's the first part of the creation of the twitter producer

4_Producer Part 1 - Writing Twitter Client_1&2
09:16
5_Producer Part 2 - Writing the Kafka Producer
13:44

this lecture represents an introduction of producer configurations

6_Producer Configurations Introduction
03:02

In this lecture we will talk about acks and min.insync.replicas properties

7_acks & min.insync.replicas
09:40

In this lecture we will talk about some important properties like retries and max.in.flight.requests.per.connection

8_retries & max.in.flight.requests.per.connection
03:25

In this lecture we explain how to produce and have a high throughput by using idempotent property

9_Idempotent Producer
06:02

This lecture represents how to have a safe producer with an example

10_Producer Part 3 - Safe Producer
09:31

in this video we will explain producer compression

11_Producer Compression
07:15

This lecture talk about producer batching

12. Producer Batching
07:32

This lecture is a hands-on example to have a high throughput producer

13. Producer Part 4 - High Throughput Producer
05:39

This lecture explains the Producer Default Partitions and Key Hashing

14. Producer Default Partitions and Key Hashing
03:35

This is an advanced lecture that explains the meaning of and how to use max.block.ms and buffer.memory

15. [Advanced] max.block.ms and buffer.memory
04:07

In this lecture, we will refactor to basically isolate the dependencies of Kafka basics and Kafka producer Twitter, into separate files that can be independently taken.

16_Refactoring the Project
06:09
+ Module 7 :Kafka ElasticSearch Consumer & Advanced
8 lectures 01:03:07

Consumer and advanced configuration overview

1. Consumer and Advanced Configuration Overview
00:37

In this lecture we will how to setup a cluster elasticsearch in the cloud

2. Setting up ElasticSearch in the Cloud
06:52

In this, we will explain the most basic to know how elasticsearch works.

3. ElasticSearch 101
12:12

In this we will do a hands-on example to consume some example data and get this data in the elasticsearch cluster

4. Consumer Part 1 - Setup Project_1
09:28

In this lecture we will continue the previous example but this time we will consume twitter data with our cluster.

4. Consumer Part 1 - Setup Project_2
11:56
5. Consumer Part 2 - Write the Consumer & Send to ElasticSearch_1
08:31
5. Consumer Part 2 - Write the Consumer & Send to ElasticSearch_2
06:34

In this lecture we will talk about delivery semantics in depth

6. Delivery Semantics for Consumers
06:57