What is Kafka Connect?

Stephane Maarek | AWS Certified Solutions Architect & Developer Associate
A free video tutorial from Stephane Maarek | AWS Certified Solutions Architect & Developer Associate
Best Selling Instructor, Kafka Guru, 9x AWS Certified
4.7 instructor rating • 36 courses • 534,234 students

Lecture description

Understand what is Kafka Connect, the problems it solves in the Kafka world

Learn more from the full course

Apache Kafka Series - Kafka Connect Hands-on Learning

Kafka Connect - Learn How to Source Twitter Data, Store in Apache Kafka Topics & Sink in ElasticSearch and PostgreSQL

04:23:34 of on-demand video • Updated September 2020

  • Configure and run Apache Kafka Source and Sink Connectors
  • Learn concepts behind Kafka Connect & the Kafka Connect architecture
  • Launch a Kafka Connect Cluster using Docker Compose
  • Deploy Kafka Connectors in Standalone and Distributed Mode
  • Write your own Kafka Connector
English [Auto] High in this lecture we're going to learn about Kepcher connect. Why can't you connect in a bit of his history. So in 2013 there was Kavkaz your points and it included a bunch of new features such as topic reapplication LA complexion. It simplified a lot. The producer client API then came Kavkaz 3.9 in November 2015. The new simplified high level consumer API without this news is our dependency was a new feature it added security with encryption authentication. And finally most importantly for this course it added the keffiyeh connects API. In his first version when May 2006 and arrived half cat 10 was out and keffiyeh kitten was out broths Cafcass streams API but also as part of the tender wanted to revisions improve the connected size Embra the single message Tristram's API. So basically a gift to connect has been around for over a year and a half now. This time recording this video and it's been continuously improving this year and a half time really allowed a lot of companies a lot of programmers to just get on board and develop some nice stuff to. We're going to use. So we have to connect. Why. Because dreams basically when you have health care you have for common use cases you have a source and you push it to. And that's the producer API. You have a topic in Kafka and you want to create another topic and Casca from its debts the consumer and the producer API. Finally you have a topic in karega and you want to place it in a sink in a Target store. That's a consumer API. And then finally just Similarly you have kaffiyeh and you want it to be consumed by your app. There's also the consumer API. So keffiyeh connect where does it fit. Well caf could connect source API. We're going to see this on the first case calfskin X API allows you to easily have a source and puts all its data into Casco then because it serves the purpose to do transformations on character topics. More importantly can't connect sync serves the third purpose into getting data out of Kafka wherever you want. And for the last use case the consumer API is still great so we keep it this way. So Kaffee connect what does it do. It helps you simplify improve getting data in and out of Kafka and we're going to see there's a lot of Connectors for you connect that I like to do that. So it connects solves problems for programmers just like you and I. Basically we've always wanted to import data from the exact same sources and I'm going to show you a list right here. Databases JD see couch space. Golden Gate is a P block chain. Cassandra Dinon Woods Eubie FCPA OOTB mongered B Mtukudzi rethink the cells for solar Eske use Twitter and so many more. Basically the number of technologies that have your source data is pretty much limited. There none of it. Number of sources are all pretty much the same or they could get grouped. Also we always want to store data into the exact same Sink's So as three elastics search HFS first a cluster or JTB database I say Rehana documents to be Casandra dynamo D.B and so on. You see all these different sort of Oulds different things. So basically with Kefaya it would seem that it's really tough to achieve full tolerance exactly once distribution ordering and so on and all these things. And some people already wrote these things for you. Other programmers may have done a really good job at writing these sources and sinks producers and consumers. And that's why I have to connect to connect. Is basically a set of connectors and all of these I've put on this slide are existing connectors that allow you for example to get data from your database straight into Africa and then that data from the database and look at how we're going to put it into say elastics search. So keffiyeh connectors really don't rewrite code that someone or is it written use someone else connecter and bring your own configuration and that's what we're going to see. And that's to paraphrase if you connect. I hope you're excited. I'm really excited about showing you how to use it. You can enter for this course and more importantly we'll show you how to use the Twitter source connector elastics search sync connector and the JTB sync connector. Finally we'll write our own connector in this class so you can get really hands on and do whatever you want.