Building serveless distributed analytics application on AWS

Name: Building serveless distributed analytics application on AWS
Rating: 5.0 (5 reviews)

Building serverless app from A to Z on AWS Cloud

Created byTomasz Stachlewski

Last updated 1/2024

English

What you'll learn

Learn how to build data-processing application on AWS
Learn how to use BigData frameworks on EMR Serverless.
Learn how to visualize and orchestrate the flow of actions using Step Functions.
Use the concept of event driven processing to automate the invocation of our processing flow thanks to EventBridge.
Build your own Data Catalog with the help of Glue
Analyze collected data using SQL queries in fully serverless way with help of Athena.
Discover the power CodeWhisperer – AI coding companion which helped to write our own PySpark scripts.

Course content

1 section • 16 lectures • 3h 11m total length

What you will build in this course?7:41
Welcome to our course! This will be a special one during which you will learn many things. It’s not one of those boring courses where you just listen about products. No! Here you will actually build something! Let's see what!
Quick Intro to EMR Serverless service5:18
EMR is the most important component/service of our application – it allows us to distribute analysis across a fleet of servers. In this lecture we will discuss the serverless option of this service – where we don’t need to think about servers at all.
Creating your first EMR Studio6:17
EMR Studio is a cloud IDE tool that makes it easy to work with big data frameworks such as Spark. In this lecture we will spin up our first Studio – which will be our central place of analysing spark jobs from serverless cluster.
Preparing Dataset9:19
Our goal is simple: we want to build a data processing flow/app – which will convert and agregate data. But to do it – we need to have some kind of data. In this lecture we will spend a few minutes to understand the dataset which will be used by us during this course. We will also create S3 bucket and define structure of files which we will be storing there.
Submiting your first jobs to EMR Serverless20:16
It’s time to actually play with EMR Serverless – in this lecture we will define our first EMR Serverless application – which we will then use to run sample spark script. Ready, Stady, Go!
AWS Step Functions Visual workflows for distributed applications4:17
AWS Step Functions – a magical service for visual creation of workflows for distributed applications. Next to EMR, this will be the most important service for our application!
Creating visual definition of our distributed application18:06
It’s start to create our flow – so let’s jump into Step Functions and lets define our application flow!.
Amazon EventBridge – building event-driven applications at scale6:12
Lets talk about EventBridge - a serverless service that uses events to connect application components together, making it easier for you to build scalable event-driven applications.
Automating the data analysis process14:35
Automation is a key! Time to integrate S3 and StepFunctions with a help of EventBridge.
Adding Data Converting step to our orchestration engine26:22
We are doing progress – it’s time to connect our Step Function flow with EMR application and convert new uploaded datasets into different format with help of Spark!
Adding Data Aggregation step to our orchestration engine12:56
Let’s try to extend our application with a bit more complex Spark Job!
Interacting with S3 from our data flow process12:28
We have added all-important EMR activities to our flow – time to add activities which will operate on S3.
Building Data Catalog and querying data with AWS Athena21:03
Athena is severless service which make it easy to write SQL queries in order to analyze data collected on S3. Let’s see how it looks in practice!
Data discovering with GenerativeAI15:10
We have started from EMR Studio and we are back here! We will play a bit with Jupyter Notebooks where we will write PySpark scripts to discover collected data. To make it easier we will use Code Whisperer - AI-powered productivity tool!
Easy cost optimization for EMR Serverless6:53
Everything works. Nice! But maybe there is still some place for improvement… ?
You made it! Congrats!4:09
Let's see what we have learned.

Requirements

No prior knowledge required!
Own AWS account will be required.

Description

Welcome in this hands-on course, during which you will build your own data-analysis application on AWS Cloud! This course is dedicated for you independently if you already know AWS or you are just starting with cloud! Together we will build step by step fully operational application for analyzing data! Each lecture will extend our application with new functionalities and features - so that at the end we will have complex and advance application!

Let's see exactly what kind of topics we will cover in our hands-on course. Only here you will:

learn how to distribute analysis across fleet of servers using EMR Serverless.
learn how to visualize and orchestrate the flow of actions using Step Functions.
use the concept of event driven processing to automate the invocation of our processing flow thanks to EventBridge.
create Spark jobs for data conversion and aggregation and run it on BigData platform.
build your own Data Catalog using Glue service.
analyze collected data using SQL queries in fully serverless way with help of Athena.
discover the power CodeWhisperer – AI coding companion which helped to write our own PySpark scripts.
optimize the cost of your application and start using ARM processors for your BigData jobs

Ready, Stady, Go! Lets build!

Who this course is for:

For all those who want to learn how to build serverless data anaysis applications in AWS Cloud!