Amazon MSK Crash Course

Explore infrastructure as code with the AWS Cloud Development Kit, and provision a VPC, a Fargate cluster, and an application load balanced elastic container service.

Explore how infrastructure as code models architecture in human readable text, serving as the source of truth and documentation, enabling auditable history and repeatable provisioning via automation.

Shows deploying a network and bastion host stacks, retrieving and running setup scripts from an S3 bucket, and installing jq, Java OpenJDK 11, Kafka, and AWS CLI for later steps.

Discover the cloud development kit, an object-oriented abstraction over CloudFormation that enables infrastructure as code in familiar languages like Python, JavaScript/TypeScript, Java, and Go, reducing context switching.

Discover the CDK’s two components—the node-based CLI for initializing, synthesizing to CloudFormation, and deploying infrastructure, and the TypeScript construct library of cloud resources—with JSII bindings across languages.

Understand the CDK's three construct types—application, stack, and resource constructs—and see how a single application hosts multiple stacks with resource constructs like ECS task definitions and CloudWatch alarms.

Explore the AWS IaC abstractions hierarchy from REST APIs to CloudFormation and the Cloud Development Kit, and learn how level 1–3 constructs accelerate secure, multi-resource deployments like microservices.

Learn to start and bootstrap an AWS CDK project, install the CDK, and deploy a Python FastAPI app with a hello endpoint and health check.

Learn how to create a new CDK project in Python, provision a VPC with ECS Fargate, deploy a hello world API behind an application load balancer, and retrieve the endpoint.

Deploy a hello API on AWS with a CDK project and CloudFormation outputs, using an application load balancer and ECS task with a container image asset and health check endpoint.

Learn how to clean up a deployed AWS CDK application by running cdk destroy to delete infrastructure and minimize charges, a process that takes about five to ten minutes.

Explore the high level deployment models and operational characteristics of the MSC service, and examine costing to level set for deeper topics later.

Learn the high level architecture of a Kafka cluster, including ZooKeeper, brokers, topics and partitions, and how replication provides fault tolerance for producer–consumer data flow, focusing on AWS MSK.

We examine how MSK deploys across two availability zones in the Amazon cloud network. AWS hosts the Zookeeper and broker nodes in its AWS-only VPCs, with ENIs routing traffic.

Compare provisioned and serverless deployment models for AWS MSK, detailing compute, storage, auto scaling, authentication, networking, and cost implications.

Explore Amazon MSK broker configuration items, including ACL behavior, default settings versus Kafka defaults, automatic topic creation, replication and minimum in-sync replicas, and unclean leader election for data safety.

Explore MSK integrations with AWS Lambda as a producer or input source, and connect to Glue Schema Registry, Kafka Connect, MSK Replicator, Kinesis Data Firehose, and managed Apache Flink.

Present base lab network options and a bastion host setup within a two-availability-zone VPC, then contrast with a production three-AZ architecture featuring public, private, isolated subnets and multiple NAT gateways.

Learn to deploy a base AWS network and bastion host with CDK, including a two-AZ VPC, single NAT, and Kafka-ready access via a bastion host.

Deploy the network and bastion host, retrieve CloudFormation outputs to copy setup scripts from S3, make them executable, and install jq, Java, Kafka, and update the AWS CLI.

Explore IAM-based authentication models for AWS MSK, provisioning clusters with IAM authentication, and integrating them with client, producer, and consumer applications.

Deploy a private two-broker AWS MSK cluster with IAM authentication; attach IAM roles to EC2 or ECS, inject temporary credentials, and enable read, write, create topics, and describe consumer groups.

Configure a provisioned MSK cluster with IAM authentication using the CDK, including a bastion host, VPC, Kafka brokers, and CloudWatch logs, and export bootstrap brokers to SSM parameter store.

Provision an IAM-enabled AWS MSK cluster with two availability zones, brokers, log groups, and bootstrap endpoints via CDK, and inspect its configuration, encryption in transit and at rest, CloudWatch monitoring.

Discover IAM authentication with MSK by running an ECS Fargate Kafka client that fetches credentials from its task role and SSM Parameter Store, then publishes and consumes greetings messages.

Configure IAM authentication for an MSK cluster from a bastion host using the AWS CLI, then create Kafka topics and verify with CLI producers and consumers.

Explore Python clients with IAM-based authentication for an MSK cluster using AWS MSK IAM SASL Python library. Cover Python 3.8 requirements, bastion updates, and CDK deployment of IAM auth stack.

Demonstrates using IAM authenticated Python clients on Amazon MSK from a bastion host, creating a chat messages topic and running a producer and consumer with SASL/SSL OAuth bearer.

Deploy an IAM-enabled spring kafka app on ECS Fargate via a CDK infra, fetching MSK bootstrap URLs from SSM parameter store and creating topics on startup.

Deploy a Spring Kafka app on ECS Fargate using IAM authentication and SSM params, and verify via ECS/CloudWatch logs that producer and consumer connect and exchange messages.

Provision an msk cluster in sasl scram authentication mode and use client applications to integrate, understanding the authentication mode and its security constraints for Kafka connections.

Discover how SASL/SCRAM authentication with MSK works by associating secrets containing usernames and passwords, validating credentials via Secrets Manager, and securing producer and consumer traffic.

Walks through provisioning a SASL/SCRAM enabled MSK cluster using CDK, including Fargate, VPC, KMS key, Secrets Manager, and bootstrap broker configuration.

Provision a sasl/scram msk cluster, inspect the aws console resources, retrieve the sasl secret and bootstrap urls, and verify encryption, monitoring, and two-az deployment for client apps.

Connect to a SASL/SCRAM MSK cluster from a bastion host using AWS CLI and Secrets Manager to create a greetings topic and exchange messages with Kafka CLI.

Learn to use a python-based kafka client to connect to an MSK cluster with SASL SCRAM authentication from a bastion host, including producer and consumer examples.

Deploy a Spring Kafka application on ECS Fargate that authenticates to a SASL SCRAM MSK cluster, using CDK, Secrets Manager, and Parameter Store for credentials and bootstrap servers.

Deploy a Spring Kafka service on ECS Fargate with SASL/SCRAM authentication against an MSK cluster, provisioning resources, and validating admin, producer, and consumer workflows.

Provision a fully managed MSK Kafka Connect instance or set of workers, and learn how Kafka Connect fits under the MSK umbrella to move data to and from Apache Kafka.

Explore the MSK connect architecture in AWS, including connectors, workers, and tasks that source from or sink to Kafka. Learn how plug ins in S3 enable deployment within private subnets.

MSK Connect is the managed version of Kafka Connect deployed in your VPC with IAM authentication, offering provisioned or autoscaled workers and a per MCU hour price.

Teach how to implement change data capture from Aurora PostgreSQL using Debezium and Kafka Connect, streaming WAL changes through MSK into Kafka for real-time processing.

Deploy Aurora Postgres with logical replication and a replication slot to enable Debezium change data capture, then connect it to the MSK cluster via IAM auth in this cdk-based setup.

Deploy a change data capture setup by provisioning an Aurora Postgres database, creating Debezium roles and the e-commerce schema, and preparing CDC on MSK with Debezium.

Deploy the Debezium PostgreSQL CDC connector for Aurora in MSK by creating topics, configuring IAM-based authentication, and deploying the CDK stack with a Debezium plugin, worker config, and connector setup.

Deploy a Debezium Postgres CDC connector on MSK Connect, configure the e-commerce CDC worker, and verify producer details in CloudWatch, including a snapshot of existing customers into the Kafka topic.

Explore deploying a Debezium Postgres CDC connector with Aurora, and understand change data capture events (schema and payload) from the initial snapshot to real-time streaming into Kafka.

Redeploy a Debezium PostgreSQL CDC connector on MSK Connect by reusing an existing worker config ARN and revision, and monitor offset storage topics during deployment.

Learn MSK connect quotas, including custom plugins and worker configurations, and how offset storage supports restarting connectors across VPCs with transit gateways or peered networks.

Explore MSK cluster operations and common administrative tasks for provisioned MSK clusters. Serverless offerings reduce these tasks, highlighting why this section matters for provisioned environments.

Scale MSK broker storage from 1 gb to 16 tb using manual or application auto scaling, within a six-hour window and 10–80% storage targets.

Deploy an MSK cluster with storage auto scaling using CDK, set min 1 GB and max 100 GB, attach an application auto scaling target, and verify in the console.

Set up a bastion host and Kafka tooling to generate heavy data into an MSK cluster, triggering storage autoscaling and verifying disk usage and broker scaling.

Watch how CloudWatch triggers a target-tracking alarm when disk utilization passes 20% and auto scales MSK broker storage, with deployment notes on updating volume size for redeploys.

Scale your MSC brokers by adding nodes with the update broker count API, understand the soft 30-broker limit, and rebalance partitions across new and existing brokers.

Scale an MSK cluster from two to four brokers across availability zones, reassign partitions so each broker handles equal shares, after creating topics with six partitions and publishing test data.

Deploy an initial two-broker amazon msk cluster with a cdk project, set up a bastion host, and expand brokers with partition reassignments using iam, mtls, or scram.

Set up a bastion host, publish scaling data to the MSK cluster, and create topics A and B with six partitions while enabling IAM authentication.

Expand an MSK cluster by adding a third broker, update the CloudFormation parameter, deploy, and monitor CloudWatch metrics to ensure partitions are balanced across brokers.

Corrects a broker count mistake during cluster expansion. Updates the broker count from three to four to satisfy two availability zones and redeploys the expanded MSK cluster.

Expand an Amazon MSK cluster from two to four brokers and reassign partitions to balance topic partitions across all four brokers using a partition reassignment plan and Kafka CLI.

Provision a Glue Schema Registry and integrate it with Kafka clients to manage the evolution of our data contracts.

Discover Glue schema registry (GSR) for central discovery and evolution of data contracts, enabling governance and loose coupling with compatibility rules including none, backward, forward, and full variants.

Explore Glue schema registry compatibility modes—none, disabled, backward, forward, full, all—and how they handle optional and non-optional fields and what versions consumers can read.

Understand glue schema registry quotas: ten registries per account and region, 1000 schema versions per region, a 170 KB payload limit, and ten metadata key-value pairs; limits can be increased.

Publish and consume Kafka messages using a Glue Schema Registry client, validating and registering schemas, serializing payloads to bytes, and performing compatibility checks with both producer and consumer.

Define a json schema for a chat message, generate a pojo with json schema to pojo, and integrate spring kafka with glue schema registry using a reusable producer/consumer library.

Explore how to build a Spring for Kafka producer and consumer with MSK and Glue Schema Registry, using JSON schemas and a CDK deployed infrastructure.

Explore building a spring kafka consumer for MSK using glue schema registry and json schema, configuring a consumer factory, listener, and json-to-pojo mapping with Jackson.

Walks through deploying Spring Kafka producers and consumers on AWS MSK with Glue Schema Registry, including CDK provisioning, schema registration, and schema versioning with backwards compatibility.