From 0 to 1: Hive for Processing Big Data
4.2 (598 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
5,321 students enrolled

From 0 to 1: Hive for Processing Big Data

End-to-End Hive : HQL, Partitioning, Bucketing, UDFs, Windowing, Optimization, Map Joins, Indexes
4.2 (598 ratings)
Course Ratings are calculated from individual students’ ratings and a variety of other signals, like age of rating and reliability, to ensure that they reflect course quality fairly and accurately.
5,321 students enrolled
Created by Loony Corn
Last updated 1/2018
English
English [Auto]
Current price: $69.99 Original price: $99.99 Discount: 30% off
5 hours left at this price!
30-Day Money-Back Guarantee
This course includes
  • 15.5 hours on-demand video
  • 137 downloadable resources
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
Training 5 or more people?

Get your team access to 4,000+ top Udemy courses anytime, anywhere.

Try Udemy for Business
What you'll learn
  • Write complex analytical queries on data in Hive and uncover insights
  • Leverage ideas of partitioning, bucketing to optimize queries in Hive
  • Customize hive with user defined functions in Java and Python
  • Understand what goes on under the hood of Hive with HDFS and MapReduce
Course content
Expand all 87 lectures 15:15:26
+ You, Us & This Course
1 lecture 02:02

We start with an introduction. What is the course about? What will you know at the end of the course? 

Preview 02:02
+ Introducing Hive
4 lectures 43:30

Data warehousing systems - which have become the rage with the rise of 'Big Data' -  are quite different from traditional transaction processing systems. Hive is a prototypical data warehousing system. 

Preview 12:59

Hive is built atop Hadoop, and can even be characterized as the SQL skin atop Hadoop MapReduce.

Hive and Hadoop
09:19

Hive tries really hard - and mostly succeeds - at pretending to be a relational DBMS, but really, under the hood its quite different - understand how, and understand schema-on-read.

Hive vs Traditional Relational DBMS
13:52

Now that we understand the differences between Hive and a traditional RDBMS, the differences between HiveQL and SQL will seem a lot less annoying and arbitrary.

HiveQL and SQL
07:20
+ Hadoop and Hive Install
5 lectures 54:31

Before we install Hive, we need to install Hadoop. Hadoop has 3 different install modes - Standalone, Pseudo-distributed and Fully Distributed. Get an overview of when to use each

Hadoop Install Modes
08:32

How to set up Hadoop in the standalone mode. Windows users need to install a Virtual Linux instance before this video. 

Hadoop Install Step 1 : Standalone Mode
15:46

Set up Hadoop in the Pseudo-Distributed mode. All Hadoop services will be up and running! 

Hadoop Install Step 2 : Pseudo-Distributed Mode
11:44

If you are all set with Hadoop, let's go ahead and install Hive. 

Hive install
12:05

Let's run a few basic queries on Hive. Head on over to the SQL primer section at the end of the course, if you have no previous experience in Hive.

Preview 06:24
+ Hadoop and HDFS Overview
2 lectures 18:25

What exactly is Hadoop? Its origins and its logical components explained.

Preview 07:25

HDFS based on GFS (The Google File System) is the storage layer within Hadoop. It stores files in blocks of 128MB. 

HDFS or the Hadoop Distributed File System
11:00
+ Hive Basics
11 lectures 01:45:27

Let's cycle through primitive datatypes in Hive.

Preview 17:07

Hive has some really cool datatypes - collections that make it feel like there is a real programming language under the hood. Oh, and btw - there is!

Collections_Arrays_Maps
09:28

Structs and unions are yet another bit of Hive that seem more at home in a programming language.

Structs and Unions
05:57

Let's get into the nitty-gritty - starting with creating tables. Remember schema-on-read? 

Create Table
13:15

Inserting into tables has a few quirks in Hive, because, after all, all writes are just data dumps that know nothing about the schema

Insert Into Table
12:04

More on inserts - remember that no schema checking happens during database writes!

Insert into Table 2
06:51

Alter table works in Hive - understand how.

Alter Table
07:22

Hive data is stored as files on HDFS, the distributed file system that is an integral part of Hadoop. Understanding the physical layout of hive tables will make many advanced concepts - bucketing and partitioning - far more clear.

HDFS
09:25

Learn how to interact with HDFS. This comes in handy if you want to understand what's going on under the hood of your Hive Queries. 

HDFS CLI - Interacting with HDFS
10:58

Let's create a few tables and see how to insert data. We'll see external tables as well and what happens under the hood in HDFS with each of these activities. 

Code-Along: Create Table
09:54

Hive CLI allows you to run scripts and execute queries directly from the command line rather than the hive shell. 

Code-Along : Hive CLI
03:06
+ Built-in Functions
4 lectures 34:28

Hive has a whole bunch of useful functions available out-of-the-box. This is an introduction to the 3 types of functions available. Standard, aggregate and table generating functions.

Preview 06:45

The case-when statement is very useful to populate columns by evaluating conditions. Size() and Cast() are other useful built-in functions.

The Case-When statement, the Size function, the Cast function
10:09

explode() is a very interesting table generating function which expands an array to produce row for every element in the array.

The Explode function
13:06
Code-Along : Hive Built - in functions
04:28
+ Sub-Queries
5 lectures 46:03

Sub-queries in Hive are rather quirky. For instance, union is fine, but intersect is not. 

Preview 07:13

Sub-queries have a few rather arcane rules - no equality signs, and some rather specific rules on exists and in.

More on subqueries: Exists and In
15:13

It is possible to insert data into a table using subqueries - just don't try to specify any schema information!

Inserting via subqueries
05:23
Code-Along : Use Subqueries to work with Collection Datatypes
05:56

Views are an awesome bit of functionality in Hive - use them. Oh, btw, views are non-materialized, if that means anything to you. If not - never mind!

Views
12:18
+ Partitioning
7 lectures 51:38

Indices are just a lot less important in Hive than they are in SQL. Understand why, and also how they can be used.

Indices
06:40

Partitioning in Hive is conceptually similar to Indexing in traditional DBMS - way to quickly look up rows with specific values in a particular column

Preview 06:36

Let's understand the why of partitioning

The Rationale for Partitioning
06:16

Partitioning needs to specified at the time of table creation - understand the syntax.

How Tables are Partitioned
09:52

Once a table has been partitioned appropriately, using it is not a lot of work.

Using Partitioned Tables
05:27

Inserting data into partitioned tables can be a bit tedious - understand how dynamic partitioning can help!

Dynamic Partitioning: Inserting data into partitioned tables
12:44

Let's see partitioning in action! 

Code-Along : Partitioning
04:03
+ Bucketing
5 lectures 48:01

Bucketing is conceptually quite close to partitioning - and indeed to Indexing in traditional RDBMS - but with a key difference. 

Preview 11:56

Bucketing has an important advantage over partitioning - the metastore is unlikely to be taken down by it.

The Advantages of Bucketing
04:54

Bucketing needs to specified at the time of table creation - understand how.

How Tables are Bucketed
12:36

Once a table has been bucketed, using it is not that difficult.

Using Bucketed Tables
07:22

Sampling is a very handy technique in a data warehouse, and bucketing helps power this functionality

Sampling
11:13
+ Windowing
4 lectures 49:39

Windowing functions start to get at the real number-crunching power of Hive. In effect, they help tack on a new column to a query result - and that column contains the results of aggregate functions on a window of rows.

Windowing Introduced
12:59

Let's use windowing to set up a running total, aka a cumulative sum, for revenues in a sales table

Windowing - A Simple Example: Cumulative Sum
09:39

Let's now make that running sum reset each day - combining the power of windowing and the power of partitioning

Windowing - A More Involved Example: Partitioning
11:54

Rownumber, rank, lead and lag - Hive places really nifty windowing functions at your disposal. 

Windowing - Special Aggregation Functions
15:07
Requirements
  • Hive requires knowledge of SQL. If you don't know SQL, please head to the SQL primer at the end of the course first.
  • You'll need to know Java if you are interested in the sections on custom user defined functions
  • No other prerequisites: The course covers everything you need to install Hive and run queries!
Description

Prerequisites: Hive requires knowledge of SQL. The course includes and SQL primer at the end. Please do that first if you don't know SQL. You'll need to know Java if you want to follow the sections on custom functions. 

Taught by a 4 person team including 2 Stanford-educated, ex-Googlers  and 2 ex-Flipkart Lead Analysts. This team has decades of practical experience in working with large-scale data. 

 Hive is like a new friend with an old face (SQL). This course is an end-to-end, practical guide to using Hive for Big Data processing. 

Let's parse that 

A new friend with an old face: Hive helps you leverage the power of Distributed computing and Hadoop for Analytical processing. It's interface is like an old friend : the very SQL like HiveQL. This course will fill in all the gaps between SQL and what you need to use Hive. 

End-to-End: The course is an end-to-end guide for using Hive:  whether you are analyst who wants to process data  or an Engineer who needs to build custom functionality or optimize performance - everything you'll need is right here. New to SQL? No need to look elsewhere. The course  has a primer on all the basic SQL constructs, . 

Practical: Everything is taught using real-life examples, working queries and code . 

What's Covered: 

Analytical Processing: Joins, Subqueries, Views, Table Generating Functions, Explode, Lateral View, Windowing and more

Tuning Hive for better functionality: Partitioning, Bucketing, Join Optimizations, Map Side Joins, Indexes, Writing custom User Defined functions in Java. UDF, UDAF, GenericUDF, GenericUDTF,  Custom functions in Python,  Implementation of MapReduce for Select, Group by and Join

For SQL Newbies: SQL In Great Depth

Who this course is for:
  • Yep! Analysts who want to write complex analytical queries on large scale data
  • Yep! Engineers who want to know more about managing Hive as their data warehousing solution