Data Lake in AWS - Easiest Way to Learn

Explore how Amazon Kinesis ingests, buffers, and processes streaming data in real time with low latency. Compare video streams, data streams, Firehose, and data analytics to build scalable real-time applications.

Configure billing alerts and delegate access to billing information for authorized users, enable free tier usage alerts, and set up CloudWatch and AWS budget alarms to monitor estimated charges.

Explore Amazon S3 storage classes by creating a bucket, uploading files, and using standard, infrequent access, intelligent tiering, deep archive, and glacier retrieval options to compare costs and retrieval times.

Protect your centralized data lake with a mix of resource-based and user-based policies. Enforce access by job roles, enable versioning, MFA, and cross-region replication in S3.

Enable S3 versioning to track and manage multiple object versions, read a specific version, delete and undelete objects, and understand storage costs for all versions.

Configure S3 lifecycle rules to enforce a 365-day retention for current versions and 30-day retention for previous versions in versioned bucket, and clean up incomplete multipart uploads after 7 days.

Set up cross-region S3 replication for disaster recovery by enabling versioning on source and destination, creating buckets in different regions, and configuring a replication rule with an IAM role.

Walk through configuring an AWS Glue service role, creating a crawler, and building a Glue catalog for S3 data, then query with Athena on the iris_csv table.

Analyze nearly 97,000 Amazon reviews in parquet format with Athena, create a table, and explore sentiment using Amazon Comprehend, examining star rating and sentiment correlations, and visualize results in Excel.

Explore AWS Glue's serverless ETL, comparing Python Pandas dataframes for small data with Spark dataframes for large datasets, and using dynamic frames and bookmarks for incremental processing.

Convert iris csv to parquet using the Glue visual editor, enabling serverless Spark, automatic Glue catalog updates, and quick Athena queries on S3 data.

Explore analyzing a university rankings dataset from Kaggle in a data lake using Athena, performing ELT and ETL with Spark, and visualizing insights with Amazon QuickSight.

Set up a Glue crawler for ranking data and query it with Athena. Use QuickSight for dashboards and address data quality, switching from lazy serde to open csv serde.

Address OpenCSVSerDe data mapping issues by creating the table with a script and validating results in Athena on S3; use string type for all columns and cast at query time.

Learn data cleanup in AWS Athena by casting strings to numbers, handling nulls with try and coalesce, and using split and regex replace for clean ranks and counts.

Perform ETL on the university ranking data using PySpark in AWS Glue, clean and transform with Spark SQL, and load results back to S3 as CSV via a notebook.

Leverage Amazon QuickSight to visualize university rankings by year and region, using Spice data import, pivot tables, and calculated fields to reveal top ranks, distinct counts, and dashboards.

Learn to cancel an AWS QuickSight subscription through a hands-on lab, practicing subscription management within a data lake context.

Learn how to manage schema changes in a data lake using the glue catalog with csv format, focusing on Athena’s index-based access and risks from shuffled or missing columns.

Learn how Athena reads Parquet by name, supports index-based access, and handles shuffled, missing, or new columns without query changes, noting that renaming columns or data type changes require testing.

Learn how csv schema changes impact athena queries and how to preserve column order, add new columns at the end, and keep the glue catalog in sync with crawlers.

Organize s3 data lake with precise crawler paths and region and time partitions to map each path to a single table, enabling athena queries and applying data classifications for security.

Partition data in S3 and create partitioned tables to query specific partitions, using iris setosa, versicolor, and virginica files to demonstrate partitioning, catalog updates, and performance gains.

Organize iris data using hive compatible partitions and a self documenting folder structure by type. Create a hive partition crawler, run MSCK and add partitions, and update the Athena catalog.

Explore Amazon Comprehend in a live lab to perform sentiment analysis, custom classification, and medical text extraction, while analyzing syntax, key phrases, and language detection.

Launch a SageMaker notebook instance, configure an IAM role for S3 access, and open Jupyter in a cloud Python development environment, then stop and restart as needed.

Clone the course repository into your sagemaker notebook, open Jupyter, and use the terminal to run git commands; pull updates to keep the code current.

Learn to use Python in a Jupyter notebook on AWS SageMaker, with pandas and numpy to read csv data, clean NaNs, analyze Iris, and visualize with plots.

Build a serverless customer review pipeline that accepts reviews, detects sentiment with AWS Comprehend, and stores transformed data in an S3 data lake using Firehose and Lambda.

Explore the six key advantages of cloud computing with AWS, including consumption-based pricing, scalability, speed, global reach, and cost-management tools.

Explore AWS global infrastructure, including public, gov, and China clouds, and see how regions and availability zones, linked by the AWS backbone, ensure data locality and low latency.