Streaming Data Pipeline from Confluent Kafka to AWS MYSQL DB

Name: Streaming Data Pipeline from Confluent Kafka to AWS MYSQL DB
Rating: 4.2 (2 reviews)

Hands on course to build Real Time Data Streaming Pipeline from Confluent Kafka to Lambda to AWS MySQL RDS from scratch

Created byNilesh Shukla

Last updated 5/2025

English

English [Auto],

What you'll learn

Hands On course to learn to use Confluent Kafka & AWS Cloud Platforms
Learn basics of navigating & using AWS Cloud Platform services like Lambda, RDS-MYSQL, IAM, Cloudwatch, Secrets Manager & more
Learn basics of Confluent Platform and overview of services like: Topics, Connectors, API Keys, Clusters & more
Watch & Learn to build on your AWS + mConfluent Accounts- Real Time Data Streaming pipeline from Confluent to AWS RDS using Serverless Lambda Compute

Coding Exercises

This course includes our updated coding exercises so you can practice your skills as you learn.

Course content

6 sections • 14 lectures • 1h 40m total length

Introduction & High Level Architecture of end product that we will build5:09
Explore the high-level architecture of a real-time streaming data pipeline from Confluent Kafka to AWS MySQL DB, built with AWS Lambda, Secrets Manager, and IAM, using Python 3.13.

Create Free Confluent.io account to use for data streaming using Kafka5:00
Create a free Confluent account to explore data streaming with Kafka, decoupling systems and feeding downstream applications like Lambda and DynamoDB for real-time processing.
Test your Knowledge: Confluent - Basics
Confluent: Create Topic, API Keys & Explore Confluent Kafka Topic capabilities5:21
Create a Confluent Kafka topic, configure partitions, retention, and schema options, explore topic capabilities, and generate API keys to enable Lambda-based data flow from the topic to AWS.
Test your knowledge: Confluent Basics
Confluent: Setup Datagen Source Connector to produce data Kafka Topic6:32
Set up a Datagen source connector in confluent cloud to populate products Kafka topic, configure API key, select product schema, and monitor data flow across partitions before pausing.
Test your knowledge: Confluent Connectors

AWS: Create & Explore Lambda Serverless Compute9:41
Explore AWS Lambda, a serverless compute service, covering its event-driven, auto-scaling, cost-efficient model, and walk through creating a function from scratch with Python 3.13, and monitoring via CloudWatch.
Test your knowledge: Lambda Serverless compute
Configure Lambda to consume from Confluent Kafka Topic & verify in CloudWatch10:16
Configure a lambda to trigger from a Confluent Kafka topic, authenticate with Secrets Manager, and verify message ingestion in CloudWatch logs, then parse records and write to MySQL.
Create & Explore AWS RDS MySQL Database8:48
Create and explore an AWS RDS MySQL database, review relational AWS options, configure a free-tier instance with VPC and security groups, and prepare to connect using MySQL Workbench.
Test your knowledge: AWS RDS MySQL Database
Connect to AWS MySql DB via MySQL Workbench & create Schema/Table7:11
Connect to an AWS MySQL database with MySQL Workbench and create a schema and a products table. Define id, name, description, and price, then insert sample data.
Test your knowledge: CloudWatch

Parse event and extract fields and write to Cloudwatch10:58
Parse the event from Confluent in Lambda, extract ID, name, description, and price from records, decode the value, and print the parsed data to CloudWatch, then insert into AWS MySQL.
Parse & Extract data from JSON - Simple
Parse & Extract data from Nested JSON - Mid Complexity
Parse & Extract data from JSON - Challenge - Array of json elements
Update Lambda to connect to RDS + Create Layer and add pymysql module to it8:12
Configure a lambda in the VPC with MySQL RDS, add subnets and security groups, and update its execution role. Create and attach a pymysql layer, deploy, and validate connectivity.
Update Lambda Python code to insert extracted fields into MySQL RDS8:46
Connect a Python lambda to a MySQL RDS using environment variables, build an insert for extracted fields into the products table, and verify results via CloudWatch logs.

Execute End to End flow and observe at all Intermediate steps3:36
Validate the end-to-end data pipeline from the source connector through Kafka, CloudWatch, and RDS MySQL by observing a sample data point and rising counts in the database, confirming end-to-end flow.
Summary, Wrap Up and Next Steps!5:51
Summarize the end-to-end streaming pipeline from Confluent Kafka to aws MySQL, highlighting lambda-based json parsing, rds mysql integration, Secrets Manager, iam roles, and CloudWatch logging.

Requirements

Be able to create AWS Cloud Platform Account (Free Tier)
Be able to create Confluent Kafka Platform Account
Basic understanding of Python

Description

In this hands-on course, participants will follow along step-by-step to build a real-time streaming data pipeline that sends data from Confluent Kafka to AWS Lambda and finally into an AWS RDS MySQL Database. This course is designed to provide practical, real-world skills by walking through each component of the architecture, ensuring that learners not only understand the concepts but also apply them directly in a cloud environment.

On the AWS side, participants will gain experience working with several important services, including AWS IAM for securing resources, AWS Secrets Manager for managing sensitive credentials, and AWS CloudWatch for monitoring and logging the data pipeline in action. These services are essential for building secure, scalable, and reliable applications in the cloud.

On the Confluent Kafka side, learners will set up a fully managed Kafka cluster, create topics for message streaming, and configure a fully managed Source Connector to simulate real-world data ingestion. This gives participants valuable exposure to enterprise-grade Kafka infrastructure without the overhead of managing the platform themselves.

By the end of this course, participants will have built a working, scalable pipeline, gained insights into cloud-native architectures, and acquired hands-on experience that can be directly applied to real-world projects or professional roles.

This course is suitable for budding Cloud Engineers, mid level Data Engineers, Product Owners, Product Mangers, Scrum Masters and Technology Leaders looking to get a hands-on experience of building a Real time Streaming Data Pipeline.

Who this course is for:

Beginner Cloud Developers looking to build Real Time Data pipeline development skills and add a real life project to resume
MId Level to Sr. Cloud Developers to quickly gain understanding of building Real Time Streaming Data Pipelines using AWS + Confluent Platforms
Product Owners looking to build understanding of capabilities, risks & issues in building data pipeline using AWS + Confluent Kafka platform
Product Managers looking to build understanding of capabilities, risks & issues in building data pipeline using AWS + Confluent Kafka platform
Technology Leaders looking to deepen understanding of capabilities, efforts, risks & issues in building data pipeline using AWS + Confluent Kafka platform
Solution Architects looking for building hands-on Data Pipeline using AWS MySQL DB + Confluent Kafka Platforms

Streaming Data Pipeline from Confluent Kafka to AWS MYSQL DB

What you'll learn

Explore related topics

Coding Exercises

Course content

Introduction1 lecture • 5min

Confluent related Activities3 lectures • 17min

AWS Account creation & Exploration1 lecture • 5min

AWS + Confluent Integration4 lectures • 36min

Parse the event & extract data fields3 lectures • 28min

End to End Data Pipeline execution & Validation2 lectures • 9min

Requirements

Description

Who this course is for: