Ultimate Guide to Data Streaming with AWS Kinesis

Name: Ultimate Guide to Data Streaming with AWS Kinesis
Rating: 4.2 (494 reviews)

Learn to build event driven systems and insights based on real-time analytics

Created byAdam McQuistan

Last updated 11/2022

English

Japanese [Auto],

What you'll learn

Collect, Process and, Analyze Data in Real-Time with AWS Kinesis
Real-Time Data Analytics
Kinesis Data Streams
Kinesis Data Firehose
Kinesis Data Analytics for SQL
Kinesis Data Analytics for Apache Flink
Pub/Sub Message Processing
Harness the Power of Kinesis Stream Processing in both Java and Python

Course content

4 sections • 54 lectures • 7h 9m total length

Course Introduction0:26
Why You Need this Course3:08
Course Overview3:22
About the Instructor2:11
Course Prerequisite Knowledge1:19

Pub/Sub Messaging: The Underlying Technology2:32
Kinesis Data Storage: Streams, Shards and, Records6:36
Shard Limits5:23
Kinesis Data Streams Cost Model3:08
Creating Kinesis Data Stream in AWS Console5:23
Creating Kinesis Data Stream with AWS CLI6:00
Overview of Producers and Consumers8:52
Producing Records to KDS with PutRecord and PutRecords7:56
Java Demo: Producing Records Individually to KDS with AWS SDK16:02
Java Demo: Producing Individual Records with Guaranteed Ordering using AWS SDK12:06
Demonstrates producing single, ordered records to a Kinesis data stream with the Java SDK put record API, using seller id as the partition key and sequence numbers for ordering.
Python Demo: Producing Records Individually to KDS with AWS SDK12:09
Demonstrates producing individual records to a Kinesis stream with the Python boto3 SDK using put_record, including a unique order payload and partition keys for shard distribution.
Python Demo: Producing Individual Records with Guaranteed Ordering using AWS SDK10:42
Learn how to write individual records to a Kinesis stream using Boto3 with guaranteed ordering via a sequence number and the seller_id partition key.
Java Demo: Producing Records In Batches to KDS with AWS SDK10:01
Python Demo: Producing Records In Batches to KDS with AWS SDK8:16
Use the Python boto3 sdk to batch and publish json-encoded records to a kinesis data stream via put records api, with 20-item batches and order id as partition key.
Introducing the Java Based Kinesis Producer Library5:17
Java Demo: Producing Records with Java and the Kinesis Producer Library8:11
Consuming Records from KDS5:44
Java Demo: Consuming Records from KDS with AWS SDK14:07
Python Demo: Consuming Records from KDS with AWS SDK9:37
Learn to consume records from a Kinesis data stream with python and boto3, listing shards, creating shard iterators, and using get_records in a continuous poll loop.
Problems of GetRecords based SDK Consumers4:51
Introducing the Java Based Kinesis Client Library8:28
Java Demo: Consuming Records with Java and the Kinesis Client Library22:51
Java Demo: Inspecting DynamoDB Lease Table used by KCL Application2:38
Inspect the DynamoDB lease table used by the Kinesis client library to checkpoint for each shard, showing one record per shard and the last processed sequence number.
Introducing AWS Lambda Consumer10:57
Java Demo: Setting up AWS SAM Project for Lambda Based Consumer13:34
Java Demo: Serverless Lambda Consumer Code Walkthrough and Deployment19:29
Java Demo: Serverless Lambda Enhanced Fan-Out Consumer10:43
Python Demo: Setting up AWS SAM Project for Lambda Based Consumer13:24
Python Demo: Serverless Lambda Consumer Code Walkthrough and Deployment15:32
Python Demo: Serverless Lambda Enhanced Fan-Out Consumer12:19
Metrics and Monitoring5:29

Kinesis Data Firehose Intro and Objectives5:52
Kinesis Data Firehose Delivery Stream Capacity1:52
Kinesis Data Firehose Code Model0:51
Demo: Transferring Raw Order Data Stream to S310:46
Create a Kinesis data stream, feed ordered data, and connect it to a Kinesis Data Firehose delivery stream to export raw orders to Amazon S3 for long-term storage.
Kinesis Data Firehose Transformations with AWS Lambda4:30
Demo: Transforming Order Delivery Stream and Sending Order Totals to S317:20
Demonstrates a Kinesis data firehose transformation using a lambda to compute order subtotals from order items and write totals to S3 via a SAM deployment.

Importance of Real-Time Data Analytics3:13
Introducing Kinesis Data Analytics4:15
Discover how Kinesis Data Analytics offers a managed serverless platform for real-time streaming data processing, including streaming SQL and a managed Apache Flink runtime with Zeppelin notebooks and Kafka.
Kinesis Data Analytics Cost Model1:49
Kinesis Data Analytics for Streaming SQL Key Concepts6:29
Kinesis Data Analytics Architecture2:36
Kinesis Data Analytics for Streaming SQL Time Aspects4:12
Kinesis Data Analytics for Streaming SQL Tumbling Windows2:44
Kinesis Data Analytics for Streaming SQL Sliding Windows5:00
Kinesis Data Analytics for Streaming SQL Staggered Windows4:05
Demo: KDA for Streaming SQL to Calculate Average Revenue per Seller Each Minute18:02
Learn to build a Kinesis Data Analytics streaming SQL application to compute seller revenue over 30-second tumbling windows, using boto3 to create streams and a SQL script to output trend.
Introducing Apache Flink8:43
Demo: Interactive Development using Zepplin Notebook18:23
Demonstrates building an interactive Apache Flink SQL app in the Kinesis Data Analytics Studio notebook, creating input/output streams, AWS Glue metadata, and a 30-second tumbling window for seller revenue.

Requirements

Basic understanding of AWS Cloud (AWS CLI, S3, CloudWatch, IAM)
How to program in either Java or Python

Description

Real-time streaming technologies are growing in popularity among the many technological drivers of business innovation because users are increasingly demanding personalized experiences which adapt and respond to them based on their journey through digital products and services. The AWS Kinesis suite of stream persistence and processing services have come to be recognized as first class choice for achieving the kinds of event driven architectures feeding into real-time analytics.

In this course students learn to harness the power of Kinesis Data Streams (KDS) and Kinesis Data Firehose (KDF) to construct high-throughput, low latency, pipelines of data across a variety of architectural components leading to scalable and loosely coupled systems. Additional focus is placed on how these stream persistence technologies are used in conjunction with Kinesis Data Analytics to perform advanced, real-time, computations which drive informed business actions and insights.

The course goes beyond the theory of what these services are, making heavy use of demonstrations and code walkthroughs to give examples of how these technologies are used in practice. Most code examples are demonstrated in parallel using both the Python and Java programming languages in an effort to reach the largest audience of developers. However, some examples are presented only in one language in cases where either one language doesn’t support a particular functionality or is significantly less complex to demonstrate.

Who this course is for:

Software Developers, Software Engineers, Data Engineers and Data Analysts interested in Data Streaming in AWS Cloud
People with strong desire to learn real-time streaming analytics and event driven systems with AWS Kinesis

Ultimate Guide to Data Streaming with AWS Kinesis

What you'll learn

Explore related topics

Course content

Introduction5 lectures • 10min

Kinesis Data Streams31 lectures • 4hr 58min

Kinesis Data Firehose6 lectures • 41min

Kinesis Data Analytics12 lectures • 1hr 20min

Requirements

Description

Who this course is for: