What is Kafka Streams?

Stephane Maarek | AWS Certified Solutions Architect & Developer Associate
A free video tutorial from Stephane Maarek | AWS Certified Solutions Architect & Developer Associate
Best Selling Instructor, Kafka Guru, 9x AWS Certified
4.7 instructor rating • 37 courses • 574,934 students

Lecture description

Learn what is Kafka Streams at a high level

Learn more from the full course

Apache Kafka Series - Kafka Streams for Data Processing

Learn the Kafka Streams API with Hands-On Examples, Learn Exactly Once, Build and Deploy Apps with Java 8

04:48:46 of on-demand video • Updated October 2020

  • Write four Kafka Streams application in Java 8
  • Configure Kafka Streams to use Exactly Once Semantics
  • Scale Kafka Streams applications
  • Program with the High Level DSL of Kafka Streams
  • Build and package your application
  • Write tests for your Kafka Streams Topology
  • And so much more!
English [Auto] High and will come to this bet you can use new course this course is on Cafcass streams. I'm glad you join me on this course and get ready. It's going to be lots of learning. So first of course introduction we're going to get started with the streams we're going to have to run the first application we guess. What is Cafcass stream's understand how it fits in the ecosystem et cetera et cetera. So with this brief introduction I really hope you can get a take away from it and the reason what we're going to do for the rest of the course. So first question is what is Cafcass streams Cafcass streams is an easy data processing and transformation library within Cathe. It ships were the Kafka binary. It's within Cafcass projects so it's not an external library created by a third party. So here you have Kafka and you can create Cafcass applications of any kind. It could be to transform data. It could be to enrich data to perform for example fraud detection or monitoring and the alerting. So there's lots of basically applications. The idea is that Cafcass streams is a library that you said on top of Kafka and that you create your application on. So what is that because streams really it's a standard Java application. It's just a Java library and you just launch it like any java application and we'll see this during the course. You don't need to create a cluster forecaster's application like you would for Sparc or fling quantify and I'll have a lecture that goes over the difference. But the easy thing is that it's just a job application. No clusters. It's highly scalable. It's elastic and fault tolerance because it inherits every specific benefit that Kafka provides because it's integrated with Kafka and that makes you really really awesome. It has exactly ones capabilities and there is a section in this course about what exactly once means. But this is the first library in the world that provides streaming exactly one's capabilities Tatane with Cafcass. And that's a huge thing in the streaming world. It processes record one at a time so there's no batching. So this is true streaming some of the libraries like sparks streaming process things in batches and then it works for any applications. So if you have a small project or a very very large project you write the same code you've got the same application and skills the same way. So it's really awesome. So let's look at the architecture design. OK. So you've seen that slide if you looked at my calf connect course but let's get over this again. So you have a cat cluster and it has several brokers OK in this case four but it can be from one to 100 or whatever you want. And you have your sources and usually the ways you are onboard you sources in the Perfect's Kafka architecture design is that you have a connect cluster and if you don't know what kind of cluster is I recommend you look at my connect course you can find a link in the last lecture of this course. So you have your corporate resources and you connect cluster basically on boards it onto Kefaya and now your data is in Kafka and you want to process it. That's where you have your streams applications or Cafcass trims application basically sit on the right hand side and they do from Kafka to Kafka. And that's really cool because all the data processing all their transformation is tightly integrated with calfskin. Finally you want to expose this transform data to your source your Sink's for example a database elastics search or whatever. Then you again use your connect cluster for this and this is all described in my connect course. So in the connectors you saw the left hand side. And in this course we're really going to see the right side to do data transformation and processing using Cafcass streams. So a bit of history about Cafcass streams this API was introduced as part of Cafcass 0 10 which was sometimes in 2016 and has been fully mature as part of Kavkaz your 10 11 which is June 2017. So this is a really new library. Again the API can change it and it will change but what are you learning here is still a very applicable in case of any changes. As I said before the only library that can leverage D-New exactly one's capability from Kavkaz you're 11 and have a whole section on this and then it is a serious contender to other streaming precessing frameworks which has sparked faling Gordon high. Or any other streaming library. So really really good to get to learn it and I'm glad you're taking this journey with me. And then finally as I said it's a new library so it's prone to change. So don't be afraid if things change in the future what you need to learn is the ideas behind it the API and all the change will be somewhat miners in the future.