Serverless Data Engineering - AWS Glue, Kinesis, Lambda

Name: Serverless Data Engineering - AWS Glue, Kinesis, Lambda
Rating: 4.9 (11 reviews)

Batch & Stream Processing using AWS Serverless Technologies

Highest Rated

Created bySoumyadeep Dey

Last updated 12/2025

English

What you'll learn

AWS Glue Batch and Stream Processing
AWS Kinesis to ingest streaming data into AWS Ecosystem
Role of AWS Lambda to process data in a pipeline
Apache Spark
AWS Glue Workflow to Orchestrate Data Pipeline in Glue

Course content

10 sections • 102 lectures • 17h 34m total length

Data Lifecycle & Data Engineering8:12
Data Engineering Technology Mapping4:42
AWS Serverless Offerings for Data Engineering3:07
Serverless benefits & About the Course7:55

Distributed Compute & Storage, Big Data, MapReduce34:11
Big Data Ecosystem3:53
File Format - Parquet & Avro20:18
Intro to Spark, Batch & Stream Processing27:21
Spark Ecosystem, Development, Architecture & Execution23:55
Spark Installation23:15
Spark Standalone Cluster in Laptop12:36
Spark on AWS EMR21:53
Spark Application Deployment Modes (--deploy-mode=cluster | client)3:11
DataFrame, DAG, PySpark Module16:56
What is SparkSession18:52
DataFrame Reader & Writer12:34
DataFrames, Partitions & Tasks5:41
Transformations - Select, Distinct, Where, Filter17:27
HANDS-ON : Spark Actions15:12
Explore Spark UI8:05
Explore the spark ui to inspect a spark application's driver and two executors, its jobs, timeline colors, and environment settings on a standalone cluster.
Transformation - agg(), alias(), groupby()14:07
Transformation - orderBy(), selectExpr()9:13
Shuffle Process in groupBy17:05
Input Partitions10:43
Output Partitions11:39
Exploring Spark UI Further15:36
Read Spark EXPLAIN Plan Output7:37
Spark DATE Functions9:54
Prepare EMR to execute Spark Applications10:19
Submit Spark code to EMR from Laptop8:43

Introduction to AWS Glue7:56
Data Catalog & Glue Crawler5:40
HANDS-ON : Use Glue Crawler to read S3 files and Store Metadata in Data Catalog14:44
HANDS-ON : Glue Connections for Other Data Sources by Crawler15:16
Glue Serverless ETL Job Configuration9:13
Visual ETL Part 1 (No Code Transformation) - Configure S3 Data Sources10:12
Visual ETL Part 2 (No Code Transformation) - Add Spark Transformations14:00
Visual ETL Part 3 (No Code Transformation) - Transformation Output8:08
HANDS-ON - Execute PySpark Code in AWS Glue11:46
Glue API Introduction7:03
HANDS-ON : Glue API - Load MySQL Table from S3 Files11:45
PySpark Database & DWH Operations from Glue7:32
HANDS-ON : S3 to MySQL using PySpark on Glue7:30
VPC Pre-requisite for Redshift Operations2:03
HANDS-ON : Glue Connection & Crawler for Redshift7:34
HANDS-ON : ETL using Glue API - MySQL OLTP to Redshift OLAP12:08
ETL using PySpark on Glue - MySQL to Redshift8:41
HANDS-ON : ETL using PySpark - MySQL OLTP to Redshift OLAP20:02
HANDS-ON : PySpark to Redshift WITHOUT Glue API3:55
Glue Workflow - ETL Job Orchestration14:33

Introduction to Kinesis Data Streams (KDS)7:18
Kinesis Streams (KDS) - Components & Capacity14:07
HANDS-ON : Create KDS6:18
HANDS-ON : Configuration - Capacity Modes, Warm Throughtput9:16
Send Data to KDS - put_record & put_records9:26
HANDS-ON : Set up Python Boto3 Environment4:30
HANDS-ON : Send Data to KDS7:51
Kinesis Producer Library (KPL)7:53
Kinesis Shard Management - Resharding9:27
Stream Consumers, Enhanced Fan-out, Shard Iterator, Python SDK Implementation8:44
HANDS-ON : Read Records from KDS using Python SDK & Multi-threading14:30
Kinesis Client Library (KCL), Agent, CloudWatch Metrics7:41

Intro to Serverless Compute & AWS Lambda Use Cases9:48
Lambda Function & its Components8:01
HANDS-ON : Configure Lambda Function & its Components16:53
Lambda Code Execution Model11:51
HANDS-ON : Write Python Code from Lambda Console, 'Event' data12:39
HANDS-ON : S3 Trigger -> Read File -> Copy to /tmp -> Print data9:00
HANDS-ON : S3 Trigger -> Read File -> Process -> Write to S3 Folder10:15
HANDS-ON : Create ZIP Deployment Package for Python Code9:48
HANDS-ON : Deploy Additional Python Packages & Modules using ZIP in Lambda10:43
HANDS-ON : S3 Trigger to Database & Concurrent Lambda Execution8:37
Architecture using Lambda3:17

Requirements

AWS Basics like EC2, IAM, VPC and S3

Description

Data Engineering! It is a big word in today's tech world. Every organization is running behind data engineers and as per analysis one of the jobs that would be untouched by AI wave. It is also one of the highest paying jobs in tech world. There are multiple technologies, frameworks, tools that facilitate Data Engineering stack. Spark, Kafka, Lakehouse, Data Warehouse etc. are some of those.

All Cloud Service Providers have complete Data Stack up their sleeve. AWS, Azure, GCP provide multiple platforms to work on data stack. One interesting stack is Serverless, a term coined by AWS with the launch of AWS Lambda.

This course focuses specifically on Serverless Data Processing, which has gained a lot of momentum over the years. AWS is the top contender in the serverless world that includes batch & stream processing. This course entails the following topics:

Serverless Batch Processing Part 1 - AWS Glue
Streaming Ingestion - Kinesis Data Streams, Kinesis Firehose
Serverless Stream Processing Part 1 - AWS Glue & Kinesis
Serverless Batch Processing Part 2 - AWS Lambda
Serverless Stream Processing Part 2 - Lambda & Kinesis

Make sure to complete the practice exercises as they will give you hands-on experience. Also, request candidates to follow along with all the hands-on labs.

Who this course is for:

Data Architects, Data Engineers who want to learn AWS Data Stack
Solution Architects who want to have AWS Data Flavour
Database Administrators (DBAs) who want to make a career shift into AWS Data
Technical Managers, Developers who want to get AWS Data Stack Overview

Serverless Data Engineering - AWS Glue, Kinesis, Lambda

What you'll learn

Explore related topics

Course content

Introduction to Serverless Data Processing on AWS4 lectures • 24min

Spark Crash Course26 lectures • 6hr 20min

Serverless Batch ETL - AWS Glue (Glue API & PySpark)20 lectures • 3hr 20min

Streaming Data Ingestion - AWS Kinesis Data Streams12 lectures • 1hr 47min

Streaming Data Storage - Kinesis (Amazon) Data Firehose5 lectures • 27min

Spark Streaming Crash Course5 lectures • 1hr 20min

Serverlss Stream Processing - AWS Glue & Kinesis7 lectures • 52min

Serverless Batch Processing - AWS Lambda11 lectures • 1hr 51min

Serverless Stream Processing - AWS Lambda & Kinesis6 lectures • 44min

Serverless Architectural Patterns6 lectures • 30min

Requirements

Description

Who this course is for: