Standalone vs Distributed Mode

Stephane Maarek | AWS Certified Solutions Architect & Developer Associate
A free video tutorial from Stephane Maarek | AWS Certified Solutions Architect & Developer Associate
Best Selling Instructor, Kafka Guru, 9x AWS Certified
4.7 instructor rating • 37 courses • 574,934 students

Lecture description

Learn about the two modes to launch Kafka Connect, Standalone mode and Distributed Mode, and their pros and cons

Learn more from the full course

Apache Kafka Series - Kafka Connect Hands-on Learning

Kafka Connect - Learn How to Source Twitter Data, Store in Apache Kafka Topics & Sink in ElasticSearch and PostgreSQL

04:23:34 of on-demand video • Updated October 2020

  • Configure and run Apache Kafka Source and Sink Connectors
  • Learn concepts behind Kafka Connect & the Kafka Connect architecture
  • Launch a Kafka Connect Cluster using Docker Compose
  • Deploy Kafka Connectors in Standalone and Distributed Mode
  • Write your own Kafka Connector
English [Auto] So you have two ways of doing give to connect workers either stand alone or in distributed mode. And we will get to try both in this course. We'll try get to try Studland first and then we'll do distribution mode for the rest of the course. So standalone mode first. Basically a single process a single worker runs all your connectors and tasks. The company curation is bundled with your process and it's very easy to get started with. It's super useful when you're doing development and testing when you're doing your own Kalka connector. It's not full tolerance. If that process fails or dies you're left without a connector. It doesn't scale or horizontally at least scale vertically by having a better view but that's it's and it's really hard to monitor because that's saying a single standalone loan process. It's very hard to monitor now distributed modes. You have multiple workers their servers basically and they run your connectors and your tasks. The configuration is not bundled with the workers. It's submitted using arrest API. And we'll see how to use our best API in details. It's super easy to scale to scale you just add workers. You just add more servers and automatically these new workers will retrieve tasks and execute them. And finally it's fault tolerance. Basically if a worker dies and we'll see in the next class if a worker dies all the tasks are rebalanced on to the available workers and your connectors can so can still go on. So it's really nice you get fault tolerance you get horizontal scalability. So all of that makes it really good really useful for production deployment of connectors. So remember central mode is made for development and testing and distributed mode is made for production deployment of connectors.