Creating Kinesis Data Firehose delivery streams8:05
Console .NET core C# entity-framework project10:54
Create a console .NET core C# app using entity framework to map grade and student models, apply code-first migrations to an RDS SQL Server database, enabling Kinesis Firehose data integration.
Kinesis Firehose C# API class6:23
Create a public Kinesis Firehose base class to populate the stream. Install the Firehose extensions and core packages, then authenticate via AWS profiles and implement batch posting with record splitting.
Kinesis Firehose C# API send data example6:30
Send json data from a code-first entity framework database to the Students firehose delivery stream using the AWS SDK for .NET, batching up to 500 records.

Creating Glue Crawlers7:09
Query Data in Athena using SQL14:56
Explore querying data in Athena with SQL against a Glue cataloged table, configure S3 query results, and optimize with Parquet via a Firehose pipeline.
Creating AWS ETL jobs16:44
Create aws glue etl jobs to extract, transform, and load from s3 into a sql server instance via athena, with vpc endpoints.
Hangfire - Automation9:25
Automate data delivery from .NET apps to Kinesis Firehose with Hangfire, using SQL Server storage and recurring jobs, while monitoring in console and storing to S3 via Glue and Athena.

Requirements

Description

The purpose of this class is to demonstrate a proof of concept using a series of lab exercise's (in the AWS Console using AWS Kinesis Data Firehose, AWS Glue, S3, Athena and the AWS SDK, with C# code using the AWS SDK) of building a Data Lake in the AWS ecosystem.
In this class, we will be sending data from a local SQL Server database to AWS RDS securely, and automatically.
I am utilizing the .NET SDK for AWS, however, this could easily be migrated to your language of choice once you understand the concepts that I am teaching.
I will be providing the full working source code of this proof of concept.
I will also provide a working example of creating Parquet files and sending to S3 (without using Firehose).

AWS developers interested in learning about building a data lake using AWS services and the AWS SDK.