
Explore AWS S3, time series data modeling, Glue ETL, Athena, and QuickSight through hands-on labs to build an enterprise IoT data dashboard project.
Explore how AWS S3 provides scalable object storage with high availability and security, and its use cases for data lakes, backups, archival, static websites, IoT, and machine learning.
Learn to create an S3 bucket in AWS, upload diverse equipment data, and organize time-series files with year-month-day folder structures for scalable analytics.
Learn about Amazon S3 key features including storage classes, lifecycle management, object lock, replication, pre-signed URLs, event notifications, strong read-after-write consistency, S3 select, batch operations, and static website hosting.
Explore AWS Glue, a fully managed, serverless data processing and ETL tool with a cloud data catalog, enabling scalable batch or near real-time data preparation across sources.
Learn key AWS Glue features for building ETL jobs in the console with notebooks and interactive sessions, including spark streaming, Glue workflows and triggers, data catalog, crawlers, and schema registry.
Explore how AWS glue crawlers populate the data catalog with metadata. Run on-demand or on a schedule, use custom classifiers for complex schemas, and produce databases and tables as metadata.
Create an AWS Glue crawler to scan S3 data, then publish a table in the AWS Glue data catalog for use with Athena.
Learn to build a glue ETL job in an interactive notebook that reads four files via the data catalog and writes the output to S3.
Discover AWS Athena, a serverless, interactive query service that analyzes data in S3 with standard SQL, built on Trino and Presto, enabling analytics across data lakes, warehouses, and on-prem sources.
Learn to query AWS data catalog tables with Athena, run queries on S3 data, and configure the query editor with a dedicated S3 bucket for results in the same region.
How Start AWS Glue interactive session.
Create a DynamicFrame from a table in the AWS Glue Data Catalog and display its schema
Convert the DynamicFrame to a Spark DataFrame and display a sample of the data which shows the 20 rows by default
Display full columns contents
Display 2 rows and full columns contents
How to use pyspark column class for dataframe column selection or creating a subset dataframe
How to add new column to spark data frame
Continuation to AWS Glue Programming 1
Write dynamic frame Dataset to AWS S3 using DataSink Object
Write dynamic frame Dataset to AWS S3 using dynamic_frame_from_options
start AWS Glue interactive session
Create a DynamicFrame from a table in the AWS Glue Data Catalog & Convert dynamicFrame to spark dataframe
How to Drops fields within a DynamicFrame
Drops all null fields in a DynamicFrame whose type is NullType
How to filter records in a dataframe
Write dynamic frame Dataset to AWS S3 using dynamic_frame_from_options
Query data using AWS Athena
How to start your AWS Glue interactive session.
How to create a dataframe from a table in the AWS Glue Data Catalog & print records count in Dataframe
How to get Distinct Rows from a DataFrame (By Comparing All Columns)
How to drops duplicate rows using dropDuplicates() function (By Comparing All Columns)
How to drops duplicate rows using dropDuplicates() function (By Comparing selected columns)
How is distinct() different from dropDuplicates()
How to do DataFrame sorting using the sort() function in Ascending Order
How to do DataFrame sorting using the sort() function in descending Order
How to do DataFrame sorting using orderBy() function (Ascending is done by default)
start AWS Glue interactive session with 2 workers
Create a Employee DataFrame using static data and display its schema & Records
Create a department DataFrame using static data and display its schema & Records
How to do Inner join between Employee & Department dataframe
How to do left join between Employee & Department dataframe
How to do right join between Employee & Department dataframe
How to do right full outer join between Employee & Department dataframe
What is a Left Semi Join with Example
What is a self Join with Example
What is a Anti Join with Example
Start AWS interactive session
Create a Static data frame
Define Encryption function
Define Decryption function
Testing Encryption & Decryption function
How to Convert a python function to pyspark udf (user define function)
How to Encrypt a dataframe column using UDF
How to Decrypt a dataframe column using UDF
Explore AWS QuickSight, a serverless, browser-based cloud business intelligence service that integrates diverse data and delivers fast insights, machine learning-powered anomaly detection and forecasting, with secure, shareable dashboards.
Build Amazon QuickSight dashboard from S3 data by creating a dataset with a manifest file, building visuals (donut chart, bar chart, and pie chart), and sharing dashboard with your organization.
Learn to build a QuickSight dashboard from multiple S3 datasets, including raw and transform data with manifest files, and publish visuals like tables, donut and line charts.
Learn to create calculated fields and calculated columns in a QuickSight data set from S3, including parsing date formats and extracting equipment attribute values with if-else logic.
Learn how to add and configure filters in a QuickSight dashboard, create visuals like pie and donut charts, apply filters to visuals, sheets, or the entire dashboard, and publish.
Create and analyze datasets in AWS QuickSight, build dashboards with charts and tables, use calculated fields and filters, and embed dashboards or visuals on a website.
Upload files to AWS S3 using dotnet eight by creating an S3 client and using the transfer utility to upload a local file to a bucket in AP South S2.
List all objects in an S3 bucket using a dotnet 8 application by fetching metadata in batches, handling continuation tokens, and iterating through results to display prefixes and sizes.
Learn how to delete a file from an AWS S3 bucket using a Microsoft .NET 8 application, including crafting a delete object request, bucket and key, and handling delete markers.
Generate a pre-signed URL for S3 objects with a Microsoft .NET 8 application to securely deliver large files to clients, reducing server CPU and memory usage, with expiry control.
Copy S3 objects from a source bucket to a destination bucket using per-object copy requests, with listing and status checks, and note S3 native copy for large datasets.
Learn how to delete all objects from an S3 bucket using a dotnet application, handling pagination with continuation tokens and verifying deletion in the console.
Learn to check if an S3 bucket or object exists using a dotnet program, by issuing get bucket location and get object metadata requests, and handle not found errors.
Learn how to list all versions of an S3 object using .NET and understand S3 versioning benefits such as restoring deleted or older versions and cheaper storage via lifecycle policies.
Learn to download an S3 object to a local machine using a .NET application by configuring the bucket name and object key, streaming the response to a file.
Discover DynamoDB, a serverless NoSQL database offering millisecond performance at any scale. Create tables with items and attributes, and use partition and sort keys along with secondary indexes and streams.
Explore DynamoDB’s serverless on-demand capacity, zero administration, and auto-scaling to zero, with NoSQL key-value and document models, strong reads, AC transactions, denormalize data models, and point-in-time backups.
Deploy a DynamoDB table using AWS CDK by bootstrapping your account, building the project, and deploying a pay-per-request table named employee detail with a string partition key.
What is AWS S3 & How to do Time series data Modelling in S3, how to upload sample data files in S3
What are AWS S3 key components & key Features (i.e S3 Object Lifecycle, Object lock, S3 replication, Presigned URL, S3 event notification, strong consistency, S3 Select, S3 Batch operations, Hosting a static website in AWS S3)
What are different storage classes in AWS S3
What is AWS Glue, it's design & key features(Spark streaming job, Glue Crawler, Data catalog & Developing AWS Glue ETL jobs , glue workflow & Triggers , aws Glue Schema Registry)
How to develop Glue ETL job in Glue studio using interactive sessions
What is AWS Athena & How to use it to run adhoc query on data in AWS S3
What is AWS Quick Sight & How to use it to create BI Dashboard
LABS on AWS S3 bucket creation, AWS Glue, AWS Athena, AWS Quick sight
Enterprise level project: which shows how to load data into S3 bucket & then crawl that data using glue crawler to create database & table in Glue catalog. use AWS glue ETL job to read & write data from S3 via data Catalog. Quick sight will be used to buid analytics dashboard from raw data & transformed data.