Apache Kafka Series - Kafka Connect Hands-on Learning
4.7 (149 ratings)
Instead of using a simple lifetime average, Udemy calculates a course's star rating by considering a number of different factors such as the number of ratings, the age of ratings, and the likelihood of fraudulent ratings.
1,416 students enrolled
Wishlisted Wishlist

Please confirm that you want to add Apache Kafka Series - Kafka Connect Hands-on Learning to your Wishlist.

Add to Wishlist

Apache Kafka Series - Kafka Connect Hands-on Learning

Kafka Connect - Learn How to use Source data from Twitter, Store in Kafka Topics & Sink in ElasticSearch and PostgreSQL
Best Seller
4.7 (149 ratings)
Instead of using a simple lifetime average, Udemy calculates a course's star rating by considering a number of different factors such as the number of ratings, the age of ratings, and the likelihood of fraudulent ratings.
1,416 students enrolled
Created by Stephane Maarek
Last updated 9/2017
English
Current price: $10 Original price: $50 Discount: 80% off
5 hours left at this price!
30-Day Money-Back Guarantee
Includes:
  • 4 hours on-demand video
  • 4 Articles
  • 3 Supplemental Resources
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
What Will I Learn?
  • You will be confident to work with Apache Kafka Connector after taking this hands-on training
  • You will learn what is Kafka Connect, Kafka Connect architecture, how to deploy a Kafka Connector in standalone and in distributed mode
  • You will learn how to launch a Kafka Connect Cluster using Docker compose
  • You will be familiar with architecture of the Kafka Connect Data Source and you will be able to deploy Kafka Connect Source Connectors
  • You will also be familiar with architecture of the Kafka Connect Data Sink and you will be able to deploy Kafka Connect Sink Connectors
View Curriculum
Requirements
  • Good understanding of Kafka Basic is REQUIRED to take this course
  • A recent Windows / Mac / Linux machine with minimum 4GB of RAM, 5 GB of disk space
  • Good to have knowledge about Linux command line
  • Good to have knowledge about Docker
  • A note book and pen is recommended to take notes during the entire course
Description

A Comprehensive and Brand New Course for Learning Apache Kafka Connect Framework with Hands-on Training – (Launched in April 2017)

Kafka Connect is a tool for scalable and reliable streaming data between Apache Kafka and other data systems. Apache Kafka Connect is a common framework for Kafka producers and consumers.

Kafka Connect offers an API, Runtime, and REST Service to enable developers to define connectors that move large data sets into and out of Kafka in real time. It inherits strong concepts such as fault-tolerance and elasticity thanks to being an extension of Kafka. Kafka Connect can ingest entire databases, collect metrics, gather logs from all your application servers into Kafka topics, making the data available for stream processing with low latency.

Kafka Connect standardises integration of other data systems with Kafka, simplifying connector development, deployment, and management.

In this course, we are going to learn the Kafka Connector deployment, configuration and management with hands-on exercises. We are also going to see the distributed and standalone modes to scale up to a large, centrally managed service supporting an entire organisation or scale down to development, testing, and small production deployments. The REST interface to submit and manage connectors to your Kafka Connect cluster via an easy to use REST API’s.

Students Loved this course – Students have rated the course with 5 stars...!!

Tin Pavlinic says “This course is an excellent resource for those getting started with Kafka-Connect. Stephane explains concepts very clearly and provides plenty of practical examples. The setup is easy. You can follow the examples along on your own machine, and then experiment further. I highly recommend this course.” (5 stars)

Alan V says "This course is GREAT: 1. The teacher provides clear explanations of the role to Kafka Connector in Kafka Streaming Solution architecture. 2. The course uses docker-compose.yml to help learners quick build a practice environment. 3. The course is well organized and easy to understand the idea, behind Kafka Connector. The teacher guides you how to use the tool step by step. 4. The provided shell commands are all well commented and steps by steps to guide users. If you read the comments in the shell script files after the course, it will remind you what the teacher just taught in the videos. 5. The teacher is very responsible to learners' questions and very helpful. If you are a professional developer, I highly recommend you to attend this course." (5 stars)

Sagar Joshi says "It was very engaging and interesting course. I recommend taking this course. Stephan have provided all the required details and its really a hands-on training course. Thank you..!!" (5 stars)

Why I should take this course?

  • My aim is to make you confident to work with Apache Kafka Connector as soon as possible by providing a hands-on training, source data from Twitter and sinking it directly into ElasticSearch and PostgreSQL database
  • In this course, you will learn what is Kafka Connect, Kafka Connect architecture, how to deploy a Kafka Connector in standalone and in distributed mode, how to setup and launch Kafka Connect Cluster using Docker compose
  • You will be able to Deploy Kafka Connect Source Connectors and Kafka Connect Sink Connectors
  • You have strong support from the instructor, I am very happy to answer all your questions and queries inside the course Q&A sections
  • You have life-time access to this course and a 30-day satisfaction guaranteed with this course

Overview of the Course Contents -

Section 1 – Course Introduction: In this section, we are going to see the required pre-requisites for this course and for Apache Kafka Connect. We are also going to talk about the objectives and the structure of the course.

Section 2 - Apache Kafka Connect Concepts: In this section, we will learn about what is Kafka Connect, Kafka Connect architecture, we will talk about Connectors, Configuration, Tasks, Workers. We are also going to learn the difference between the standalone vs distributed mode of the Kafka Connect.

Section 3 – Setup and Launch Kafka Connect Cluster: In this sections, we will learn how to install Docker on your machine and get started with Apache Kafka Connect in the simplest way possible using Docker Compose.

Section 4 – Apache Kafka Connect Data Source – Hands-on: In this section, we will gain some practical experience on Kafka Connect Data Source. We will learn the Kafka Connect Data Source architecture and the list of available connectors. We will have some hands-on practice and learning on File Stream Source Connector both standalone and distributed, and on Twitter Source Connectors…!!!

Section 5 - Apache Kafka Connect Data Sink – Hands-on: In this section, we will gain some practical experience on Kafka Connect Data Sink. We will learn the Kafka Connect Data Sink architecture, Kafka Connect REST API’s and we will have some hands-on practice and learning on Elastic Search Sink Connector and on JDBC Sink Connectors…!!!

Section 6 – Next Steps: In this section, we are going to conclude the course and going to see what is next step you can follow.

You have lifetime access to the course and 30 days’ money back guarantee, so what are you thinking? Click on “Enroll Now” button now and see you inside the course…!!!

Who is the target audience?
  • This course is for Professionals who wants to learn Apache Kafka Connect Framework and get hands-on with it
  • This course is for Professionals who are good at Apache Kafka ecosystem and basic core concepts
  • This course is for Architects who are looking into understanding Kafka Connect
Students Who Viewed This Course Also Viewed
Curriculum For This Course
50 Lectures
04:01:11
+
Course Introduction
4 Lectures 05:18

Pre-requisites for this Kafka Connect course

Preview 01:09

Learn about the course objectives:

- Learn about Kafka Connect concepts
- Install Docker and launch our Kafka cluster and Kafka Connect Cluster
- Deploy your first Source connectors
- Deploy your first Sink connectors

Preview 01:20

Learn about the course structure

Preview 01:55

Learn a bit about your instructor

Preview 00:54
+
Kafka Connect Concepts
5 Lectures 16:22

Understand what is Kafka Connect, the problems it solves in the Kafka world

Preview 04:50

Learn about the typical Kafka Connect Architecture in your enterprise

Preview 03:20

Deeper dive into Kafka Connect concepts, learn about Connectors, Configurations, Tasks and Workers

Preview 03:03

Learn about the two modes to launch Kafka Connect, Standalone mode and Distributed Mode, and their pros and cons

Preview 02:11

Visit the Kafka Connect distributed architecture and concepts in details

Preview 02:58
+
Setup and Launch Kafka Connect Cluster
8 Lectures 30:18
Important information about installation
01:01

Install Docker on the more recent Mac versions

Docker on Mac (recent versions)
03:54

Install Docker Toolbox on the less recent Mac versions

Docker Toolbox on Mac (older versions)
05:16

Install Docker on Linux

Docker on Linux (Ubuntu as an example)
04:56

Install Docker on Windows 10 64bit

Docker on Windows 10 64bit
03:42

Install Docker Toolbox on the other Windows versions

Docker Toolbox on Windows (older versions)
05:37

Note for Docker Toolbox users

Important note for Docker Toolbox users
00:30

Instructions to start our Kafka connect cluster using the Docker installation we have just created

Starting Kafka Connect Cluster using Docker Compose
05:21
+
Troubleshooting Kafka Connect
3 Lectures 09:33

Steps to troubleshoot an error

It's not working! What to do?
02:49

Where to view the Kafka Connect logs

Where to view logs?
04:05

How to ask a question in the Q&A

Asking a question in the Q&A
02:39
+
Kafka Connect Source - Hands On
8 Lectures 48:03

Looking at the Connect Source Architecture design that we will go over in this section

Kafka Connect Source Architecture Design
01:14

Running our first source connector in standalone mode!
This goes over the setup of the worker and the setup of the standalone connector

FileStream Source Connector - Standalone Mode - Part 1
09:05

Running our first source connector in standalone mode! 
This goes over the deployment of that connector

FileStream Source Connector - Standalone Mode - Part 2
07:56

Learn how to run our file source connector in distributed mode

FileStream Source Connector - Distributed Mode
11:29

Overview of the list of available connectors online, and the list of connectors available on our docker image

List of Available Connectors
03:39

Look at the configuration for the Twitter Source Connector

Twitter Source Connector - Distributed Mode - Part 1
06:53

Deploy and analyse the output of the Twitter Source Connector on our Kafka Connect Cluster

Twitter Source Connector - Distributed Mode - Part 2
06:11

Summary of what we have learned for the Section on Kafka Connect Source Connectors

Section Summary
01:36
+
Kafka Connect Sink - Hands On
5 Lectures 42:26

Architecture discussion for the sink connectors in distributed mode

Kafka Connect Sink Architecture Design
01:43

Launching our first Sink Connector in Distributed mode, to put data into ElasticSearch

ElasticSearch Sink Connector - Distributed Mode - Part 1
08:58

ElasticSearch Sink Connector - Distributed Mode - Part 2
07:31

Learn about the Kafka Connect REST API, so you can speed up and automate your connector deployments and monitoring

Kafka Connect REST API
11:08

Learn about the JDBC Sink Connector, that we will launch in distributed mode. The data will go directly into PostgreSQL

JDBC Sink Connector - Distributed Mode
13:06
+
Writing your own Kafka Connector
14 Lectures 01:26:01

Overview of the Source Kafka Connector, the GitHubSourceConnector

Goal of the section: GitHubSourceConnector
01:34

Where to download the code

Finding the code and installing required software
03:09

Description of the GitHub API

Description of the GitHub Issues API
03:55

Using the Maven Archetype to create your source and sink connectors

Using the Maven Archetype to get started
05:43

Creating our first Config Definition and define all the parameters needed for our application

Config Definitions
12:31

Analysis of the Source Connector Class

Connector Class
06:57

Writing our different schemas that will be used for keys and values

Writing a schema
07:31

Creating the Plain Old Java Objects for our data

Data Model for our Objects
06:10

Writing the GitHub API HTTP Client

Writing our GitHub API HTTP Client
07:03

Learn what Source Partition and Source Offsets are used for

Source Partition & Source Offsets
04:22

Source Tasks analysis

Source Task
13:40

Learn how to use Maven to build and run our connector

Building and running a Connector in Standalone Mode
04:59

Learn how to deploy the jars onto the Connect Cluster

Deploying our Connector on the Landoop cluster
06:30

Links to more resources available online to write your own connectors

More Resources for Developers
01:57
+
Next steps!
2 Lectures 02:18

Next steps in your learning of Kafka Connect

What's next?
01:54

Congratulations!

Congratulations and Thank you!
00:24
+
Bonus Lecture - Coupons For My Other Courses - Student Special Offer
1 Lecture 00:58

Check out my other courses and keep on learning! Coupons inside :)

Bonus Lecture: Student Special Coupons for my Other courses
00:58
About the Instructor
Stephane Maarek
4.6 Average rating
1,498 Reviews
6,758 Students
6 Courses
Best Selling Instructor, Kafka Guru, AWS Solutions Architect

Stephane is a solutions architect, consultant and software developer that has a particular interest in all things related to big data and analytics. He's also a best seller instructor on Udemy for his courses in Apache Kafka, Apache NiFi and AWS Lambda!

He loves Apache Kafka. He regularly contributes to the Apache Kafka project and wrote a guest blog post featured on the Confluent website, the company behind Apache Kafka. 

He also is an AWS certified Solutions Architect, and has many years of experience with technologies such as Apache Kafka, Apache NiFi, Apache Spark, Hadoop, PostgreSQL, Tableau, Spotfire, Docker and Ansible amongst many others. 

His favourite programming languages are Scala and Python, and he plans on learning Go soon. 

During his spare time he enjoys cooking, practicing yoga, surfing, watching TV shows, and traveling to awesome destinations!