Big Data Analysis on AWS and Microsoft Azure

Name: Big Data Analysis on AWS and Microsoft Azure
Rating: 4.2 (25 reviews)

Learn various tool for big data analysis on cloud

Created byHarshit Srivastava, Pranjal Srivastava

Last updated 5/2023

English

What you'll learn

You could create a Crawler using AWS Glue for creating a dataset and find its schema
You could perform data analysis on a dataset by running SQL queries in AWS Athena
In this course we would explore various Big Data Analytics services available on Microsoft Azure cloud.
You would learn HDInsight, and Streaming Analysis on Microsoft Azure

Course content

6 sections • 21 lectures • 1h 44m total length

Course Introduction0:29

Create a S3 bucket and add dataset4:59
Create a Crawler using AWS Glue5:14
Create an AWS Glue crawler to catalog data from S3 and other sources, producing a Glue table with inferred schema for ETL and SQL queries.
Configuring output database name for crawler3:23
Configure an AWS Glue crawler to store output in a database, specify the data source and authentication, and adjust frequency and schema options to create tables using the detected schema.
Customize Schema, find table details in Glue and Log groups in Cloud6:37

Getting started with Streaming Analytics5:22
Stream analytics is a managed serverless engine that analyzes real-time data streams from applications, devices, and blob storage. Create a stream analytics job and connect a storage account for inputs.
Connecting input with Stream Analytics job7:58
Connect blob storage input to the stream analytics job, configure input sources such as blob storage, input hub, iot hub, and event hub, and set up outputs for analysis.
Connecting output with Stream Analytics job2:07
Connect the output storage to your stream analytics job by adding a blob storage output, configuring container name and path pattern, then saving to complete the connection.
Let's write a transformation query8:44
Write a stream analytics transformation query using a sql-like language to process input json data from blob storage, configure input and output streams, and test the query with sample data.
Results1:14
Upload a single-record test input as CSV or JSON, run your stream analytics job, and download the resulting output in the specified format.
Conclusion0:36

Getting Started with Quicksight- installation5:50
Importing dataset and understanding group and values3:53
Import a dataset into QuickSight from CSV or Excel, create your BI report, and build visualizations such as pie, bar, and tree map by dropping fields into group and value.
Creating Treemap and Customizing charts4:59
Create treemap charts in Quicksight and customize existing visuals by adjusting size, color, and grouping fields, then export and share dashboards for insights.
Data Preparation- Editing Dataset before creating Charts6:06
Edit and customize your dataset in Quicksight, applying filters, creating calculated fields, and changing data types to prepare data for visualization and data cleaning.
Create a Calculated Field using Functions- ceil and concat5:43
Sankey Chart5:42
Learn to create two-layer Sankey charts in Quicksight to visualize data flows from sources to destinations, using profit, order quantity, and other numerical weights.

Requirements

You must be familiar with AWS and Microsoft Azure at fundamental level.
A basic understanding of Data Analytics concepts would be more than sufficient, however it is not required to begin with this course.

Description

In this course you will learn about various cloud services on AWS and Microsoft Azure that can be used for simple Data Analytics and Big Data Analytics. You can choose a combination of cloud resources and methods that can be integrated together to build a solution and various use cases. This course begins with lessons on Amazon Glue that is widely used by developers to build a crawler that can span through a variety of data sources and fetch some useful data from them and store those information in Amazon Athena as datasets. Later on, we can perform various types of operations and run SQL queries on Athena just any other database for finding useful insights from dataset. Moreover, you can also prepare a dataset by transforming it using SQL.

What You'll Learn:

Introduction to Cloud-Based Data Analytics: Understand the significance of cloud platforms in today's data-driven world and how AWS and Microsoft Azure stand out in this domain.
Exploring AWS Services:
- Amazon Glue: Discover how developers leverage Amazon Glue to craft crawlers that can navigate through diverse data sources, extracting valuable data. These datasets are then stored in Amazon Athena.
- Amazon Athena: Dive into the functionalities of Athena, which allows you to execute SQL queries and operations, akin to traditional databases, to derive meaningful insights from your datasets. Learn the art of data transformation using SQL.
- Amazon S3: Get hands-on experience in creating an S3 bucket, adding datasets, and understanding its integration with other AWS services.
- AWS Glue Advanced Features: Delve deeper into configuring output database names for crawlers, customizing schemas, understanding table details in Glue, and exploring log groups in the cloud.
- Practical SQL with Athena: Run SQL queries on Athena, save the output in an S3 bucket, and master the creation and saving of custom queries in AWS Athena.
Unraveling Microsoft Azure's Data Analysis Services:
- Azure HDInsight: Begin your Azure journey with practical lessons on HDInsight, Microsoft's service for big data analytics.
- Azure Stream Analytics: Transition into the realm of real-time data processing with Azure's Stream Analytics, understanding its significance in today's fast-paced digital world.

Why This Course?

Whether you're a budding data analyst, a seasoned data scientist, or someone curious about cloud-based data analytics, this course offers a structured pathway to mastering the tools and services on AWS and Microsoft Azure. By the end of this course, you'll be equipped with the knowledge and skills to make informed decisions about the right combination of cloud resources and methods, and how to integrate them seamlessly to build robust data solutions for various use cases.

Who this course is for:

Anyone who is curious to learn Data Analysis on AWS Cloud
Students and IT professionals interested in learning AWS Glue, Athena
Interested in creating a Crawler in AWS Glue to create a dataset with schema
Would be interested in learning various Big Data Analytics services and options available on Microsoft Azure.
Want to make a career in Cloud Computing and Data Analytics.

Big Data Analysis on AWS and Microsoft Azure

What you'll learn

Explore related topics

Course content

Introduction1 lecture • 1min

AWS Glue4 lectures • 20min

AWS Athena2 lectures • 9min

HDInsight on Azure2 lectures • 17min

Streaming Analysis on Azure6 lectures • 26min

AWS Quicksight6 lectures • 32min

Requirements

Description

Who this course is for: