Working with Hadoop (Dec 2022)

Name: Working with Hadoop (Dec 2022)
Rating: 4.8 (3 reviews)

Learn the Advance Features of Hadoop Ecosystem with Hands-On

Created byProton Expert Systems & Solutions

Last updated 9/2023

English

What you'll learn

Importing Incremental data from RDBMS to HDFS and from RDBMS to Hive
Hive Partitioning, Bucketing and Indexing
Exporting Incremental Data from hive to RDBMS and from HDFS to RDBMS
Creating Hive Tables for Different file formats
Developing the Pig Latin Scripts in Pig
Scheduling the OOZIE Workflow using Coordinator
Scheduling the OOZIE Sub-Workflow using coordinator
Flume Integration with HDFS
Reading Data from HDFS to Spark 1.x
Reading and Loading data from Hive to spark 1.x using spark SQL

Course content

9 sections • 24 lectures • 1h 54m total length

Introduction3:49
In this lesson, you will have information about Detailed Course Objectives, the Course Description based on the Objectives, Course Target Audience, Prerequisites for taking up this course, detailed Course Schedule and Practice Environment used in the course to do the activities.

Lesson 1: Working with SQOOP4:31
In this lesson, you will learn about Incremental Import from RDBMS to HDFS and RDBMS to Hive using Sqoop, Incremental Export from HDFS to RDBMS and from Hive to RDBMS using Sqoop. Further you will learn about Sqoop Eval Functions.
Practice 1-1: Import Incremental Data from RDBMS to HDFS and from RDBMS to Hive4:48
In this practice, you will learn how to import the incremental data from RDBMS to HDFS and RDBMS to Hive through SQOOP Import.
Practice 1-2: Export Incremental Data from HIVE to RDBMS and from HDFS to RDBMS3:56
In this Practise, you will learn how to export the incremental data from HDFS to MYSQL table and Hive table to MYSQL table.
Quiz on Sqoop Concepts

Lesson 2: Working with HIVE8:16
In this lesson, you will learn about creating dashboard in Hive, import and export the incremental data, Alter the tables, load the data into Hive tables, Executing Hive Queries for Hive built-in functions, Creation of Hive UDF’s.
Practice 2-1: Working with HQL Scripts in HIVE8:18
In this practice, you will learn how HQL queries work in hive and the commands supported by the Hive.
Quiz on Hive Concepts

Lesson 3: Data Storage and Performance in HIVE8:55
In this lesson, you will learn about creating the Hive partitioning, Hive bucketing, Hive indexing and creating the Hive tables in different file formats.
Practice 3-1: Hive Partitioning1:46
In this practice, you will learn about data partitioning of tables in HIVE using HQL.
Practice 3-2: Hive Bucketing0:52
In this practice, you will learn about Hive Bucketing in Hive using HQL.
Practice 3-3: Hive Indexing0:57
In the practices, you will learn about Indexing in Hive.
Practice 3-4: Creating Hive Tables for Different File Formats2:36
In this practice, you will learn to create hive tables for different file formats namely TEXT, AVRO, ORC, PARQUET and SEQUENCE.
Quiz on Data Storage and Performance in HIVE

Lesson 4: Working with Pig - Troubleshooting and Optimization10:59
In this lesson, you will learn about features of Pig and to create different pig scripts, execute them and use different functions to perform ETL (Export, Transform and Load). You will further learn about different Pig built-in functions, Creating Pig UDF’s, Data sampling and Debugging in Pig, Illustrate the tips for improving the performance of Pig jobs.
Practice 4-1: Developing the Pig Latin Scripts in Pig10:25
In this practice, you will learn to develop and execute different functions of pig such as Input and output (LOAD, STORE, DUMP), Eval functions (AVG, CONCAT, COUNT, DIFF, MAX, MIN, SUM, TOKENIZE), String functions (INDEXOF, STRSPLIT, SUBSTRING), Relational operators (FLATTEN, COGROUP, ORDER, UNION, SPLIT, DISTINCT) and Tuple, Bag, Map, and perform ETL operation on Pig user define functions (UDFs).
Quiz on Pig Concepts

Lesson 5: Working with Oozie7:47
In this lesson, you will learn about Oozie workflow action nodes, Scheduling the Oozie workflow and sub-workflow using coordinator file, Executing the Pig and Shell Action, Re-run a failed Job in Oozie.
Practice 5-1: Scheduling the OOZIE Workflow using Coordinator3:20
In this practice, you will learn to perform the SQOOP action by running and scheduling the Oozie workflow using coordinator.
Practice 5-2: Scheduling the OOZIE Sub-Workflow using coordinator2:35
In this practice, you will learn to schedule sub-workflow action job in Oozie.
Quiz on Oozie Concepts

Lesson 6: Integration of Flume with HDFS5:34
In this lesson, you will learn about flume and its features, architecture and models. Further explore on building blocks of flume, creating flume configuration file to import data from server directly to HDFS using flume integration with HDFS and monitoring the Cluster.
Practice 6-1: Flume Integration with HDFS1:33
In this practice, you will learn to integrate flume with HDFS to import the data from server directly to HDFS.
Quiz on Flime Integration with HDFS

Lesson 7: Cloudera Administration8:37
In this lesson, you will learn about cloudera manager and its features. Further you will learn the concepts of cloudera manager, checking logs and status of jobs in cloudera manager, create charts for monitoring cloudera manager, API access to Cloudera manager.
Practice 7-1: Creating the Dashboard in Cloudera Manager1:27
In this practice, you will learn to create dashboard in the Cloudera manager and further add charts to monitor the Cloudera cluster.
Practice 7-2: Verifying the Logs and status of Job in Cloudera Manager1:35
In this practice, you will learn how to check the job status of the task in the Cloudera manager.
Quiz on Cloudera Administration

Lesson 8: Introduction to Scala and Apache Spark8:46
In this lesson, you will learn about Scala and its features, Spark ecosystems and its components, illustrate spark SQLContext, Data units in Spark. You will get to know the working with RDD and its operation in spark with example. Further you will learn about Spark1.x and spark 2.x.
Practice 8-1: Read Data from HDFS to Spark 1.x1:24
In this practice, you will learn to read data from HDFS to SPARK1.x executed in Scala shell.
Practice 8-2: Read and Load data from Hive to spark 1.x using spark SQL1:53
In this practice, you will learn to read the data and load the data from hive to spark using spark.
Quiz on Scala and Apache Spark Concepts

Requirements

Hadoop Fundamentals (one of our courses in Udemy)
Basic Python Programming Knowledge
Working Knowledge on Data Base Systems and Data Warehouses
Basic Java Programming Knowledge
Basic Linux Commands

Description

If you are looking for building the skills and mastering in Big Data concepts, Then this is the course for you.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. In this course, you will learn about the Hadoop components, Incremental Import and export Using SQOOP, Explore on databases in Hive with different data transformations. Illustration of Hive partitioning, bucketing and indexing. You will get to know about Apache Pig with its features and functions, Pig UDF’s, data sampling and debugging, working with Oozie workflow and sub-workflow, shell action, scheduling and monitoring coordinator, Flume with its features, building blocks of Flume, API access to Cloudera manager, Scala program with example, Spark Ecosystem and its Components, and Data units in spark.

What are you waiting for?

Hurry up!

Who this course is for:

Data Base and Data Warehouse Developers
Big Data Developers and Architects
Data Scientists and Analysts
Any technical personnel who are interested learning and Exploring the features of Big Data and Tools

Working with Hadoop (Dec 2022)

What you'll learn

Explore related topics

Course content

Introduction1 lecture • 4min

Lesson 1: Working with SQOOP3 lectures • 13min

Hive Concepts2 lectures • 17min

Data Storage and Performance in HIVE5 lectures • 15min

Working with Pig2 lectures • 21min

Oozie Concepts3 lectures • 14min

Flume Integration with HDFS2 lectures • 7min

Cloudera Administration3 lectures • 12min

Scala and Apache Spark3 lectures • 12min

Requirements

Description

Who this course is for: