Creating Data Pipelines using AWS & Confluent Kafka Platform

Name: Creating Data Pipelines using AWS & Confluent Kafka Platform
Rating: 4.3 (4 reviews)

Hands on course to build Real Time Data streaming pipeline from Confluent, to Lambda to DynamoDB from scratch

Created byNilesh Shukla

Last updated 5/2025

English

English [Auto],

What you'll learn

Get an high level overview of building Data Pipelines using confluent Kafka and AWS services - Lambda, DynamoDB
Be able to build End to End Real time streaming Data Pipeline using AWS + Confluent Platforms
Build understanding of popular AWS services like Lambda, DynamoDB, IAM, Cloudwatch
Hands on practice with streaming data from Confluent Kafka all the way to Database on AWS

Coding Exercises

This course includes our updated coding exercises so you can practice your skills as you learn.

Course content

10 sections • 13 lectures • 1h 12m total length

Introduction to Real Time Streaming Cloud Native Data Pipeline Architecture2:22
This video provides High Level Overview & Architecture of the Data Pipeline and the components that we will use to build it. We will use both AWS and Confluent cloud services in this course.

Create Data Gen Source Connector, load Topic & validate6:47
In this Video Lecture, we get to action. We create data gen source connector and publish sample messages to the Topic we created in last Lecture! After observing the data in Confluent topic - we pause connector to avoid costs. We also explore some aspects of topic/offset & partition keys on Confluent Kafka Topic,
Test your knowledge: Confluent Connectors

Initial Lambda Setup, Test & Cloudwatch Exploration7:42
In this Lecture we complete initial setup of Lambda function, we print out event, test it out and look at cloudwatch logs.
Setup Lambda-Confluent connection (ESM), use Secrets Mgr & IAM role6:43
In this Video Lecture, we configure Lambda Trigger to point to Confluent Kafka Topic, before we do that, we need to store API Key and Secret in Secrets Manager & use it when configuring Lambda trigger. we then finish creating trigger and pickup testing in next video lecture.
Test Data flow from Confluent source connector to Topic to AWS Lambda Function4:22
In this Video Lecture, we leverage settings done in earlier videos and start the data source connector and watch & validate the messages from source connector flow into Confluent Topic and from then onto AWS CloudWatch via AWS Lambda Function.

Create Dynamo DB table & Explore NoSQL Capabilities5:24
In this video, we breakdown event as received in Lambda from Confluent and use python to parse out data elements. We log interim steps into CloudWatch and troubleshoot as we code along. This parsing will form basis to update DynamoDB table in next video.
Update Lambda function to insert incoming data to DynamoDB table5:19
In this Video Lecture, we update Lambda code and the Lambda execution role's IAM permissions to be able to insert data into DynamoDB as data is flowing from upstream. Lot of troubleshooting and debugging, but we finally make it to work!

Requirements

No programming experience needed, Be able to create free tier AWS and Confluent accounts for hands on practice

Description

In this hands-on, project-based course, you’ll learn how to build a cloud-native, real-time streaming data pipeline using the powerful combination of Confluent Kafka and AWS services. Designed for developers, data engineers, and technology leaders, this course takes you step-by-step through the creation of a fully functional data pipeline — from sourcing to storage — leveraging some of the most in-demand technologies in the cloud ecosystem.

You’ll start by setting up free-tier accounts on both Confluent Cloud and AWS. From there, you’ll build and configure a Kafka cluster, create topics, and use connectors to manage data flow. On the AWS side, you’ll create and configure AWS Lambda functions (Python 3.10+) to consume messages from Kafka topics, parse them, and insert them into a DynamoDB table.

By the end of the course, you will have completed an end-to-end, event-driven architecture with real-time streaming capabilities. We’ll also walk through how to monitor and verify your pipeline using CloudWatch Logs, and responsibly clean up your resources to avoid unnecessary charges. This course will help build confidence on starting to use other AWS and Confluent services and build real time streaming applications in your future or current job role. As a Leader this course will help jump start your Cloud thought process and help you understand deeper details on what goes on building cloud native data pipelines.

Whether you're a beginner, Data Engineer, Solution Architect, Product Owner, Product Manager or a Technology Leader looking to better understand streaming data architecture in action, this course provides both practical skills and architectural insights.

This course will help those who are looking to switch careers & will help to add a real life project to their Resume and boost their hands-on AWS Cloud Technical & Architecture skills. This Course will help participants upskill themselves by understanding nuances up close, of cloud native real time streaming data pipeline data pipeline setup, its issues, risks, challenges, benefits & shortcomings.

Who this course is for:

Beginner Cloud Developers looking to build AWS + Confluent Kafka Skills
Entry Level Cloud & Data Engineers looking for a Real Life Project to add to their Resumes
Mid Level and Sr. Data Engineers who are looking to enhance their skills in Confluent + AWS Platform areas
Product Owners and Product Managers looking to understand capabilities of the Confluent + AWS 's Real Time Streaming Platforms
Technology Leaders looking to understand Capabilities, Shortcomings, efforts it takes to build Real Time Data Streaming Pipeline

Creating Data Pipelines using AWS & Confluent Kafka Platform

What you'll learn

Explore related topics

Coding Exercises

Course content

Introduction1 lecture • 2min

Confluent Cloud Account Setup1 lecture • 5min

Review Cluster on Confluent & Create Topic1 lecture • 4min

Create Data Gen Source Connector and load the Confluent Topic1 lecture • 7min

Create & Setup AWS Account1 lecture • 5min

Create Lambda Function & Connect to Confluent Topic3 lectures • 19min

Parse event using Python Lambda code & write to CloudWatch Logs1 lecture • 9min

Create DynamoDB Table & Update Lambda function code to insert data into it2 lectures • 11min

Run the Data Pipeline End to End & Verify1 lecture • 6min

Summary & Wrap Up!1 lecture • 4min

Requirements

Description

Who this course is for: