What is Kafka Connect?

Right in this lecture, we're going to learn about Kafka Connect, why you have to connect in a bit of history. So in 2013, there was Cafcass zero point eight, and it included a bunch of new features such as topic replication competition, it simplified a lot to the producer client API. Then came Kavkaz zero point nine in November 2015. The new simplified, high level consumer API without this music is a keeper dependency was a new feature in added security with encryption and authentication. And finally, most importantly for this course, it added the cafÈ Connect APIs in his first version. When May 2006 and arrived, cafÈ 10 was out and Kirketon was out, Bratt's Cafcass streams API, but also as part of the 10.1 TED, the two revisions improve the connected APIs and brought the single message from the API. So basically KFKA Connect has been around for over a year and a half now at this time recording this video and it's been continuously improving this year and a half time. Really allowed a lot of companies, a lot of programmers to just get on board and develop some nice stuff to it. We're going to use so we have to connect WikiLeaks's streams. Basically, when you have KFKA for common use cases. You have a source and you push it to Kafka, and that's the producer API. You have a topic in Kafka and you want to create another topic in Kafka from its that's the consumer and the producer API finally have a topic in Kafka and you want to place it in a sink in a Target store that's a consumer API. And then finally, just similarly, you have Kafka and you want to be consumed by your app. That's also the consumer API. So Kafka Connect, where does it fit? Well, Kafka connects Source API that we're going to see this on the first case. Kafka Source API allows you to easily have a source and put all this data into COFCO. Then Clickstream served a purpose to do transformations on KFKA topics. More importantly, Coffeecake Connect Sink served the third purpose into getting data out of Kafka, wherever you want. And for the last three decades, the consumer is still great. So we keep it this way. So caffè connect, what does it do? It helps you simplify and improve getting data in and out of Kafka. And we're going to see there's a lot of connectors for Kafka to connect that. I like to do that. So Kafka connects solves problems for programmers just like you and I. Basically, we've always wanted to import data from the exact same sources. And I'm going to show you a list right here. Database's JDBC couch based Golden Gate ASEP Block chain, Cassandre Dinamo DB, FTP Iot, Mongo DB and everything, give you Salesforce, Florescu Twitter and so many more. Basically the number of technologies that have your source data. Is pretty much limited, there's not an infinite number of data sources, they're all pretty much, you know, the same or they could get grouped. Also, we always want to store data into the exact same thing. So as three ElasticSearch HDFC, a Hadoop cluster or JBC database, SAP, Hana Duckman TV, Cassandra Dinamo TV and so on. You see all these different sort of different things. So basically with Kafka, we've seen that it's really tough to achieve full tolerance. Exactly. One's distribution, ordering and so on and all these things. And and some people already wrote these things for you. Other programmers may have done a really good job at writing these sources and sinks producers and consumers. And that's why Kafka connects. Catholic Connect is basically a set of connectors. And all of these have puts on this slide, our existing connectors that allow you, for example, to get data from your database straight into COFCO and then that data from the database into kaffiyeh, we're going to put it into, say, ElasticSearch. So KFKA Connect is really don't rewrite code that someone or is it written you someone else connect her and bring your own configuration. And that's what we're going to see. And that's the power of Kefu Connect. I hope you're excited. I'm really excited about showing you how to use a few connectors for this course. And more importantly, we'll show you how to use the Twitter source Connector, the ElasticSearch Sync Connector and the JDBC Sync Connector. Finally, we'll write our own connector in this class so you can get really hands on and do whatever you want.