Serverless Data Engineering - AWS Glue, Kinesis, Lambda

Explore distributed storage and compute for big data using Hadoop concepts like HDFS and MapReduce, understand metadata and data nodes, and see how frameworks like Spark enable scalable processing.

Explore the Hadoop big data ecosystem, from HDFS and YARN to MapReduce, Hive, Pig, and Mahout, and contrast Spark, HBase, and NoSQL with data ingestion via Flume, Kafka, and Scoop.

Analyze Parquet and Avro file formats, contrasting columnar and row oriented storage, with metadata, compression, and row groups, and their use in lakehouse and Iceberg.

Explore Apache Spark as a distributed data processing framework for batch and stream processing, including PySpark basics, Spark UI, and Spark's role in ingestion and processing within data engineering pipelines.

Explore the spark ecosystem, from storage and ingestion to core apis, dataframes, and streaming. Learn spark architecture with driver and executor roles, clusters, and hosting platforms.

install spark on a local machine or ec2 by setting up jdk, optionally scala, downloading spark, configuring java and spark env vars, and enabling the history server for PySpark.

Set up a spark standalone cluster on your laptop by configuring a master and workers, spark-env, and history server, with UIs at localhost:8080, 4040, and 18080.

Explore Spark on AWS EMR: understand EMR topology, primary, core, and task nodes, Yarn resource manager, HDFS and S3 data storage, Spark Submit workflows, and secure cluster setup.

Explore spark submit deployment modes on EMR clusters: cluster mode runs the driver on a core node; client mode runs it on your local machine, notebooks defaulting to client mode.

Learn how data frames serve as the in-memory, immutable building blocks of Spark, enabling transformations and a directed acyclic graph (dag) workflow built atop RDDs, with PySpark integration.

Learn to create and configure a spark session in PySpark with builder, master, and app name, then build data frames from pandas and read streams from Kafka or Kinesis.

Read orders.csv into a Spark data frame and write it to Parquet, using header and schema options, overwrite mode, and memory considerations.

Explore how data frames, partitions, and tasks map to Spark execution, illustrating why a csv to parquet job yields 73 tasks and 73 output partitions.

Learn to transform a CSV into Spark dataframes by creating a Spark session, reading with header and infer schema, and applying select, where, filter, and distinct to shape data.

Explore spark actions and lazy loading in a hands-on workflow that drops, sorts, and filters education, vehicle, and housing loan data, then writes parquet files.

Explore the spark ui to inspect a spark application's driver and two executors, its jobs, timeline colors, and environment settings on a standalone cluster.

Use agg and alias in PySpark to compute sum, min, max, and average balances and opening dates, then group by transaction status to summarize amounts.

Sort a data frame by start date, principal amount, and end date with ascending and descending orders. Use select expr to compute interest and illustrate case statements for loan type.

Explore the shuffle process in Spark through a hands-on group by example, inspecting the Spark UI to see how four input partitions become 200 shuffle partitions and tasks.

Explore how input partitions shape spark tasks, compare csv and parquet performance, and tune spark.sql.files.maxPartitionBytes to control partition counts and processing speed.

Explore how output partitions control Spark writes, comparing repartition, coalesce, and partition by to manage shuffles, file counts, and partitioning by column values.

Explore the spark UI in depth to trace spark jobs, stages, and memory/shuffle behavior, map dag operations to code, and analyze sql data frame execution for performance tuning.

Learn how Spark translates data frame API code into a physical execution plan using explain, and how to read and compare explain output with the Spark DAG visualization.

Prepare an EMR cluster and run Spark applications with Yarn and Spark settings, configuring executor memory and cores, AWS CLI access, and S3 parquet outputs.

Submit spark code to emr from laptop using aws cli and emr steps; configure cluster id, step name, and spark submit arguments, then monitor via history server and s3 logs.

Explore AWS Glue, a serverless ETL platform that runs PySpark and Scala, covering the data catalog and crawlers, visual ETL, and workflow orchestration for batch and streaming data.

Discover the Glue data catalog as a metadata repository for tables, views, and schemas across relational, NoSQL, lakehouse, and data warehouse stores, with Glue crawler for automatic schema discovery.

Create a Glue crawler to read S3 CSV files across multiple folders, store the inferred schema in the Glue Data Catalog, and explore the generated tables.

Explore how AWS Glue functions as a metadata catalog and serverless ETL engine, configuring workers, DPUs, and Spark options to run PySpark or Scala jobs with automatic scaling.

Explore visual ETL in AWS glue for no-code data transformations. Configure multiple s3 data sources, join them via the data catalog, and output to a new s3 location.

Master no-code visual etl in aws glue studio to perform spark transformations, join five tables, rename and drop fields, detect sensitive data, and export parquet to s3.

Learn visual ETL in AWS Glue for no-code transformations, inspect the transformation output, and validate results in S3 and Athena via the Glue data catalog.

Explore how AWS Glue wraps Spark as a serverless cluster, using dynamic frames and Glue APIs to convert to data frames, read from Glue Data Catalog, and write to Aurora.

Demonstrate loading csv data from s3 into aurora mysql using glue visual etl and glue data catalog, then contrast glue api dynamic frames with pyspark for database loading.

Learn how to run PySpark in AWS Glue with a minimal code change by creating a spark context, glue context, and spark session, enabling native PySpark workflows in Glue's environment.

Learn how to use PySpark in AWS Glue for serverless data engineering to read from and write to relational databases and data warehouses via JDBC, with partitioning and driver management.

Read a csv from S3 with a Spark script using Glue context, then write to an Aurora MySQL transaction table via JDBC, with jars stored in S3.

Create a VPC interface endpoint for Secrets Manager in the same VPC as your glue job and redshift, configuring subnets, security groups, and NAT as needed to avoid errors.

Read records from an OLTP MySQL Aurora and load transformed data into a Redshift OLAP database using AWS Glue visual ETL and PySpark, after cataloging Redshift tables with crawlers.

Execute a hands-on Glue API ETL that migrates MySQL OLTP data to Redshift OLAP using PySpark, joining transaction, account, and customer tables and aligning column order.

Apply PySpark on Glue to read from Aurora MySQL, transform and load to Redshift via a Glue dynamic frame, converting the final data frame and using a Glue connection.

Apply PySpark in AWS Glue to extract MySQL OLTP data, enrich with account and customer joins to derive source and destination countries, then load to Redshift OLAP.

Learn to write from PySpark to Redshift on Glue without the Glue API, using Redshift JDBC, dependent jars, and VPC endpoint prerequisites.

Orchestrate Glue ETL jobs and crawlers with Glue workflow, using on demand, schedule, or conditional triggers and job parameters, noting Glue-only limitations.

Learn how streaming data is ingested and stored using AWS Kinesis Data Streams, a scalable service that uses partitions for near real-time processing and temporary storage.

Explore Kinesis Data Streams by understanding shards, partition keys, and sequence numbers, and how producers and consumers interact, with retention and capacity modes on demand and provisioned, and throughput.

Learn to create a kinesis data stream in the kinesis console, compare provisioned and on‑demand capacity modes, estimate shard requirements, and configure settings like retention, encryption, monitoring, and tags.

Explore configuring Amazon Kinesis data streams with on-demand and provisioned capacity, warm throughput, and shard management, plus monitoring metrics to optimize put and get operations.

Discover how to send data to Kinesis data streams with put_record and put_records, compare shard throughput limits, and learn when put_records maximizes efficiency through concise examples.

Set up a python boto3 environment to write data to Kinesis Data Streams, install boto3, configure AWS credentials, and create a Kinesis client to use put_record or put_records.

Build a Python Kinesis producer using boto3 to read 25-record files, set user-id as the partition key, and send records to the Kinesis Data Streams via Put Records API.

Explore how the Kinesis Producer Library (KPL) optimizes data writes to Kinesis Data Streams with batching, aggregation by partition key, and multi-stream capability via a Java wrapper.

Manage Kinesis data streams by resharding: split or merge shards to scale throughput, balancing on-demand auto scaling with provisioned capacity, using the update shard count API.

Read records from Kinesis data streams by using consumer applications, manage shard iterators, and implement get_records with Python boto3, including enhanced fan-out for scalable throughput.

Learn to read records from a Kinesis data stream using the Python SDK and multithreading to query multiple shards in parallel, using shard iterators and various read options.

Explore how the KCL simplifies reading from Kinesis Data Streams by handling shards, iterators, and checkpointing across consumer workers. Use CloudWatch metrics to monitor put/get throughput and iterator age.

Learn how Kinesis Data Firehose ingests streaming data from Kinesis Data Streams or Kafka and delivers to S3 data lake, Redshift, OpenSearch, Iceberg, or monitoring tools, with buffering settings.

Understand how data firehose uses lambda for basic transformations on buffered streaming data, with five-minute execution limits, and JSON to Parquet or ORC conversion in S3; CSV requires Lambda conversion.

Explore how data firehose streams data to iceberg table destinations, enabling lakehouse capabilities with open table format for row-level updates, deletes, and inserts via glue data catalog integration.

Explore end-to-end data delivery from Kinesis Data Streams to Kinesis Data Firehose using real-time ingestion and the Python SDK, with optional lambda transformation and S3 delivery.

Learn to use the Python SDK to write records to a Kinesis Data Firehose delivery stream, sending batch records to S3 with 1 MB buffers and a 300-second interval.

Explore spark structured streaming with micro-batches, triggers, and read/write stream APIs, reading from Kafka or Kinesis and writing to sinks, with append, update, and complete output modes.

Build a streaming pipeline with PySpark streaming by reading from Kafka, transforming data, and writing via write stream with processing time triggers and await termination.

Learn how event time tumbling windows demarcate data by generation time, not processing time, using the window api and group by on the event timestamp to compute aggregations.

Demonstrate event-time processing with tumbling windows in Spark streaming, aggregating stock trades by exchange and ticker to compute total buy price within defined windows using Kafka input and processing-time triggers.

Learn stream processing with AWS Glue using PySpark or Glue Studio to ingest streaming data from Kinesis or Kafka and apply Spark concepts: processing time, event time, windows, watermark, sinks.

Learn a hands-on glue streaming workflow with Jupyter notebook to process Kinesis streams. Include prerequisites, jar setup, and Spark session; write results to S3 and validate via Athena.

Prepare a Glue notebook environment by provisioning a Kinesis stream, an S3 bucket, and an IAM role with a PassRole policy, then run PySpark streaming code in the Glue notebook.

Explore implementing tumbling and sliding windows in Spark streaming with Glue notebooks, using for each batch to write to sinks like S3 or Redshift, and handle checkpointing and output modes.

Practice sliding window stream processing in a glue notebook by reading from a kinesis data stream, defining the event schema, and applying a 3-minute sliding window to produce overlapping windows.

Practice hands-on stream processing with Glue ETL using PySpark to implement a tumbling window, read from Kinesis, write to S3, and manage checkpoints and jars.

Explore how AWS Lambda delivers serverless compute for data processing, its 15-minute limit, and why it supports event-driven workloads such as S3 file processing, Kinesis streaming, and SQS data flows.

Configure a data engineering lambda in AWS: create the function, set runtime and IAM roles, connect S3 triggers, manage concurrency and VPC settings, and monitor with CloudWatch.

Understand the lambda execution model and the Python entry point concept, where a handler in lambda uses event and context data from triggers like S3, Kinesis, or Kafka.

Write python code from the lambda console or an IDE, deploy to lambda, and trigger via S3 uploads. Inspect event and context, configure the lambda handler, and test with CloudWatch.

Copy the S3 triggered file to /tmp within a lambda function using boto3, then read emp_data.csv and print emp_id and email. Troubleshoot S3 client naming and timeouts via CloudWatch logs.

Demonstrates an s3-triggered lambda pipeline that reads a csv from s3 tmp, processes to keep only employee id and email, and writes the result to an s3 output folder.

Upload a Python program to S3, create a zip deployment package with PyMySQL, and deploy a Lambda that writes S3 data into an Aurora Serverless MySQL database.

Learn to deploy python packages to AWS Lambda by building a deployment package from the virtual environment's site-packages, zipping dependencies like py mysql and boto3, and uploading.

Explore architecture using AWS Lambda at the start of the data engineering pipeline, triggering from S3, Kinesis, SQS, or Kafka and routing data to S3, databases, or Spark.

Learn how Lambda consumes Kinesis data streams in batches with configurable batch size, parallelization factor, and event source mappings, and how Kinesis filtering preserves ordering per shard.

Discover how to process Kinesis records in a Python lambda by iterating event['records'], configure batch size and window, set starting position, and apply tumbling windows and parallelization.

Set up Lambda and Kinesis integration to read from Kinesis, process with a Lambda, and store results in S3, while configuring SNS, IAM role, VPC, and triggers.

Learn hands-on how to process Kinesis stream data with Lambda, decoding base64 payloads from records, converting bytes to strings, and configuring triggers, S3 and SNS destinations, and resource settings.

Code a simple Lambda function to process Kinesis records, set the correct handler, and print data for debugging. Deploy the function, monitor invocations, duration, and logs in CloudWatch to troubleshoot.

Learn to use AWS Lambda to read from Kinesis and write to S3. Decode base64 payloads, convert to JSON, and write batched records to S3 with boto3.

Explore how Lambda architecture separates batch and stream processing with AWS Glue, Kinesis, and Lambda. Converge pipelines in a data warehouse or lakehouse for unified analytics.

Compare lambda architecture’s two-pipeline approach with kappa architecture’s streaming focus, using Kinesis and glue to unify processing and reduce maintenance.

Explore medallion architecture and how serverless AWS tools structure storage into bronze, silver, and gold layers, from raw bronze landing to lakehouse silver and reporting-ready gold.

Explore a real time IoT architecture using AWS Kinesis, Glue, and Firehose to stream, transform, and store data for dashboards with QuickSight, Power BI, Tableau.

Learn real time web application analytics with a serverless stack from api gateway to Kinesis and Glue streaming, delivering outputs to Lakehouse or DynamoDB for real time insights.

Learn the elt pattern: load into Redshift via Kinesis, transform with Glue, and reload to build fact and dimension tables for reporting.