Hadoop on Azure. An Introduction to Big Data Using HDInsight

Name: Hadoop on Azure. An Introduction to Big Data Using HDInsight
Rating: 3.4 (76 reviews)

A Pragmatic Introduction To HDInsight

Created byMike West

Last updated 12/2015

English

What you'll learn

Understand the basics concepts of Big Data.
Understand the mostly widely used tool for working with big data... Hadoop.
Learn what Microsoft is doing to help organizations create Hadoop clusters faster and more affordably than ever before.

Course content

5 sections • 32 lectures • 37m total length

Course Introduction1:18
What's the course about?

In this course we are going to cover Hadoop on Azure.

Microsoft's take on Big Data is a game changer.
The Data Explosion1:20
Approximately 90% of all organizational data is unstructured.

That means only 10% is stored in traditional relational databases.

The amount of data stored and analyzed will grow exponentially in the coming years.
What Are We Going To Cover In This Course1:49
This is a very specific course.

This course will focus on Microsoft's approach to big data.

We will be learning Azure HDInsight.
Is This Course Right For You?1:22
I want to make sure you are in the right place.

If you are looking to learn Microsoft's approach to big data then this course is for you.

This is not a traditional Hadoop course.
Terminology0:28
Let's cover some of the key terminology associated with this section.
Summary0:35
Let's wrap up what we've learned.
Quiz

What is Hadoop?1:08
Big Data is an ecosystem and Hadoop is a product.

Let's learn what Hadoop really is.
The High Level Hadoop Ecosystem1:04
There's a lot of new terminology here.

Let's learn what's involved and what the key components to Hadoop are.
HDFS - The NameNode is the Brain2:03
One of the core components of Hadoop is the NameNode.

Let's learn what it is in this lecture.
MapReduce - Scheduling and Execution1:27
MapReduce is both and engine and a programming model.

Let's learn about the map and the reduce.
PIG and HIVE1:27
The Pig programming language is designed to handle any kind of data, hence the name.

Let's learn about the two most prevalent Hadoop languages.
YARN Vernacular1:38
MapReduce has undergone a complete overhaul in hadoop-0.23 and we now have, what we call, MapReduce 2.0 (MRv2) or YARN

Let's learn about this new feature.
YARN - Separation of Duties0:21
YARN was designed for separation of duties.

In this short lecture let's look at a visual representation of YARN.
Terminology1:31
Let's learn the new vernacular introduced in this section.
Summary0:29
Let's wrap up what we've covered so far in this section.
Quiz

The Azure Data Lake's 3 Services0:48
At this juncture, Azure Data Lake is made up of three different services.

Let's learn about them in this lecture.
What's a Data Lake?2:05
The data lake concept is fairly new.

Let's learn what a data lake is and more importantly, if we need one or not.
Azure Data Lake Store1:42
Let's talk about Microsoft's Azure Data Lake in the Cloud.
HDInsight1:29
In this lecture we will learn about Hadoop in the cloud.
Azure Data Lake Analytics0:18
A new service built on Apache YARN that dynamically scales distributed infrastructure
Azure Data Factory1:21
U-SQL. The New Language For Working With Big Data2:58
Microsoft's new language for big data.
Terminology0:43
Let's cover the key terms used in this section.
Summary0:35
Let's look at some bullet points on what we've covered in this section.
Quiz

Create Azure Account0:14
In order to start our work with big data in the cloud we will need an account.

In this lecture we navigate to the URL to create one.
Create Storage and Provision Our First Cluster2:50
In order to start working with our HDInsight clusters we have on major dependency.

We need a storage account.

In this lesson we will learn how to create a storage account and provision our first cluster.
The HDInsight Management Dashboard1:51
In this lesson we will learn the basics of managing our cluster.

We will also look at the new portal... which provides us a much more granular view of our clusters.
RDP Into Cluster1:35
In this lesson let's learn how to remote into our cluster.
Terminology0:14
Let's go over the key words in this lesson.
Summary0:32
Let's go over the high points on what we've covered in this section.
Quiz

Requirements

Only the desire to learn Microsoft's direction for Big Data.
The basic concepts behind cloud architecture and SQL Server would be beneficial.

Description

Massive amounts of data are being collected on just about everything and only a small part of that data is being analyzed.

In 2014, every second over 5700 tweets were sent and 870 Facebook links were sent.

In 2013, about 4.4 zettabytes of data were created and approximately 5% of it was analyzed.

By 2020, it’s estimated that we will collect 44 zettabytes of data and the amount we analyze will jump to 40%.

One of the most overused words in recent times is “Big Data”

But what does the word really mean?

Big data refers to data being collected in ever-escalating volumes, at increasingly high velocities, and for a widening variety of unstructured formats and variable semantic contexts.

Big data describes any large body of digital information, from the text in a Twitter feed, to the sensor information from industrial equipment, to information about customer browsing and purchases on an online catalog.

Big data can be historical (meaning stored data) or real-time (meaning streamed directly from the source).

For big data to provide actionable intelligence or insight, not only must the right questions be asked and data be relevant to the issues be collected, the data must be accessible, cleaned, analyzed, and then presented in a useful way.

HDInsight is a cloud implementation on Microsoft Azure of the rapidly exanding Apache Hadoop technology stack that is the go-to solution for big data analysis.

It includes implementations of Storm, HBase, Pig, Hive, Sqoop, Oozie, Ambari, and so on. HDInsight also integrates with business intelligence (BI) tools such as Excel, SQL Server Analysis Services, and SQL Server Reporting Services.

Note: This is not a hands on course. This course creates a knowledge foundation for my next course in this series which is using what we've learned to create a real world end to end big data solution with Azure HDInsight.

Who this course is for:

You are a developer, DBA or windows admin seeking to learn how Microsoft is chaining the Big Data landscape.
You curios about Big Data and want to learn the more.

Hadoop on Azure. An Introduction to Big Data Using HDInsight

What you'll learn

Explore related topics

Course content

An Introduction To Big Data on Azure6 lectures • 7min

Hadoop High Level Overview9 lectures • 11min

HDInsight In Your Data Lake9 lectures • 12min

HDInsight on Azure6 lectures • 7min

Conclusion2 lectures • 1min

Requirements

Description

Who this course is for: