Build a Secure Data Lake in AWS using AWS Lake Formation

Name: Build a Secure Data Lake in AWS using AWS Lake Formation
Rating: 4.1 (427 reviews)

Step by step guide for setting up a data lake in AWS using Lake formation, Glue, DataBrew, Athena, Redshift, Macie etc.

Created byYomi Owoyemi

Last updated 2/2022

English

English [Auto],

What you'll learn

How to quickly setup a data lake in AWS using AWS Lake formation
You will learn to build real-world data pipeline using AWS glue studio and ingest data from sources such as RDS, Kinesis Firehose and DynamoDB
You will learn how to transform data using AWS Glue Studio and AWS Glue DataBrew
You will acquire good data engineering skills in AWS using AWS lake formation, Glue Studio and, blueprints and workflows in lake formation

Course content

6 sections • 16 lectures • 3h 18m total length

Introduction to the course11:42
Explore building a secure data lake and warehouse on AWS using Lake Formation, Glue, and Redshift, covering ingestion, cataloging, governance, and analytics-ready pipelines.

Configuring S3 lake formation8:41
Configure AWS lake formation by creating an admin in IAM, granting administrator and lake formation data admin access, and linking raw, processed, and curated S3 buckets to databases.
Simple file ingestion into the data lake13:05
Upload a csv into the raw landing zone, configure an AWS Glue crawler to catalog the data in Lake Formation, and query the resulting table with Athena.
Use blueprints and workflows in Lake formation for ingesting data from MySQL RDS17:09
Shows ingesting world_x from a MySQL RDS into a Lake Formation landing zone using AWS blueprints and workflows to automate crawlers, glue connections, and parquet format data in raw storage.
Ingest real-time data using Kinesis firehose into the data lake10:07
Set up a Kinesis firehose delivery stream to S3 to ingest real-time data into the data lake, grant Lake Formation permissions, and validate with a Glue crawler and Athena.
Security and governance of our data lake with governed tables14:21
Utilize governance tables in AWS Lake Formation to implement fine-grained, row- and column-level access controls that secure and govern data in the data lake, using tags and auditing for discoverability.

Introduction to AWS Glue DataBrew19:05
Learn to use AWS Glue DataBrew to quickly profile and clean data in an AWS Lake Formation data lake, enabling low-code data preparation, visualization, and recipe-driven transformations.
Analysis and transformation of data in our data lake with Glue DataBrew14:53
Learn to profile customer data in your data lake using Databrew, apply out-of-the-box transformations, build and manage a recipe, and preview and apply steps to refine data quality and lineage.
Create DataBrew recipes and applying them to a larger datasets10:19
Explore data transformations in databrew, including inner joins and groupings, and apply these recipes to city and country datasets to produce a combined, enriched table.

Introduction to AWS Glue Studio16:33
Learn to use Glue Studio to visually build, run, and monitor extract, transform, and load jobs, connect data sources, and populate the data catalog from raw to processed lake zones.
Author ETL jobs for moving data between the different zones in our data lake7:42
Author ETL jobs in glue studio to join city and country sources, rename fields, and publish a parquet processed city country view in lake formation.
Ingest data from DynamoDB into the data lake using AWS Glue and catalog it9:50
Create a DynamoDB table, crawl it with AWS Glue, catalog the data in Lake Formation, and ingest it into S3 as parquet, exposing a raw_movie_ingest table in Athena.

Introduction to Amazon Redshift and setting up our Amazon Redshfit cluster15:38
Explore Amazon Redshift, a scalable data warehouse with columnar architecture and massive parallel processing for fast OLAP analytics, which integrates with your S3 data lake to form a lakehouse.
Author ETL job for moving data from our data lake into the Redshift warehouse11:06
Ingest data from the data lake into the Redshift warehouse using a Glue ETL job, after configuring Lake Formation connections and crawlers to publish tables like customer.
Using Redshift Spectrum for querying data located in our data lake4:33
Query data directly in your data lake with Redshift Spectrum, avoiding ETL and data duplication, while using the Glue Data Catalog and IAM roles for secure access.

Requirements

Basic undestanding of cloud computing
Basic understanding of what a data lake and data warehouse are is essential but not required
An active AWS account is required to be able to follow along

Description

In this course, we will be creating a data lake using AWS Lake Formation and bring data warehouse capabilites to the data lake to form the lakehouse architecture using Amazon Redshift. Using Lake Formation, we also collect and catalog data from different data sources, move the data into our S3 data lake, and then clean and classify them.

The course will follow a logical progression of a real world project implementation with hands on experience of setting up a data lake, creating data pipelines for ingestion and transforming your data in preparation for analytics and reporting.

Chapter 1

Setup the data lake using lake formation
Create different data sources (MySQL RDS and Kinesis)
Ingest data from the MYSQL RDS data source into the data lake by setting up blueprint and workflow jobs in lake formation
Catalog our Database using crawlers
Use governed tables for managing access control and security
Query our data lake using Athena

Chapter 2,

Explore the use of AWS Gluw DataBrew for profiling and understanding our data before we starting performing complex ETL jobs.
Create Recipes for manipulating the data in our data lake using different transformations
Clean and normalise data
Run jobs to apply the recipes on all new data or larger datasets

Chapter 3

Introduce Glue Studio
Author and monitor ETL jobs for tranforming our data and moving them between different zone of our data lake
Create a DynamoDB source and ingest data into our data lake using AWS Glue

Chapter 4

Introduce and create a redshift cluster to bring datawarehouse capabilities to our data lake to form the lakehouse architecture
Create ETL jobs for moving data from our lake into the warehouse for analytics
Use redshift spectrum to query against data in our S3 data lake without the need for duplicating data or infrastructure

Chapter 5

Introduce Amazon Macie for managing data security and data privacy and ensure we can continue to identify sensitive data at scale as our data lake grows

Who this course is for:

Data Architects looking to architect data integration solutions in AWS cloud
Data Engineers
Anyone looking to start a career as an AWS Data Engineer
Data Scientist, Data Analyts and Database Administrators
IT professionals looking to move into the Data Engineering space

Build a Secure Data Lake in AWS using AWS Lake Formation

What you'll learn

Explore related topics

Course content

Introduction1 lecture • 12min

Setting up the Data Lake with AWS Lake formation5 lectures • 1hr 3min

Preparation and analysis of data in our data lake using AWS Glue DataBrew3 lectures • 44min

Author, run and monitor ETL jobs using AWS Glue Studio3 lectures • 34min

Prepare our data for analytics and reporting3 lectures • 31min

Bonus1 lecture • 14min

Requirements

Description

Who this course is for: