Process Big Data using Apache PIG

Name: Process Big Data using Apache PIG
Rating: 3.7 (30 reviews)

Learn analyzing and processing big data Using Apache Pig

Created byInsculpt Technologies

Last updated 3/2016

English

English [Auto],

What you'll learn

Overview of Big Data and Hadoop Framework
Anatomy of a MapReduce Framework
Basics of Apache Pig tool and Where we should use it or not
Run Pig in different Modes
Use Pig Latin Queries
Different types of PIG Operators for analysing the data
Understand the architecture of PIG tool
Work with PIG data model
Different kinds of built-in functions
Advanced PIG concepts such as PIG Streaming, PIG scripts and User Defined Functions(UDFs)
Compress the input files, final output files and intermediate output files
Pig Unit Testing, PIG Macros and Parameter Substitution
How to embed PIG in Java

Course content

8 sections • 49 lectures • 5h 41m total length

0.1 Introduction to Apache PIG1:12
0.2 Prerequisites for Apache PIG0:43
0.3 Exploring Use Cases of PIG2:42
0.4 History of Apache PIG1:46

1.1 Big Data Overview3:27
In this video we'll take a look on overview of Big Data.
1.2 Apache Hadoop Framework2:29
In this video we'll discuss about the Big Data framework.
1.3 Hadoop Distributed File System5:00
In this video we'll discuss about Hadoop Distributed File System.
1.4 Hadoop MapReduce3:33
In this video we'll discuss about Hadoop MapReduce.
1.5 Introduction to Apache Pig1:42
This is the introductory lecture for Apache PIG.
1.6 Importance of Apache PIG1:11
Apache Pig processes big data batch workloads and handles structured, semi-structured, and unstructured data, powered by open-source collaboration and a strong developer community for data manipulation.
1.7 Why PIG Over MapReduce1:50
1.8 Where PIG is Best Suited1:37
Leverage Pig Latin on Hadoop to process petabytes of weblogs and logs, enabling rapid analytics of traffic patterns and product trends, with secure access and customizable piggeries.
1.9 Where to Avoid PIG0:28
Identify where to avoid Apache Pig for big data processing of unstructured data, such as audio and raw text. Highlight how unstructured formats slow processing and prompt code optimization.

3.1 PIG Data Model- Scalar Datatype6:19
Explore the Pig data model's scalar datatypes and how loaded data gets a schema, including int, long, float, double, string, char, and byte, then load, describe, and dump a relation.
3.2 PIG Data Model - Complex DataType11:02
3.3 Arithmetic Operators6:37
Explore arithmetic operators in Apache Pig, including addition, subtraction, multiplication, division, and remainder, with practical examples using load data, a relation, and the dump command.
3.4 Comparison Operators10:44
3.5 Cast Operators12:52
3.6 Type Construction Operator7:03
3.7 Relational Operators0:48
Explore relational operators in Apache Pig, including loading, storing, joining, filtering, and combining and splitting data. Learn how these operators transform data.
3.8 Loading and Storing Operators5:38
3.9 Filtering Operators10:45
3.10 Filtering Operators-Pig Streaming with Python6:50
Learn how to perform pig streaming with Python to cube integers from a dataset by reading from standard input, streaming through a Python script, and dumping the results.
3.11 Grouping and Joining Operators- PART-116:41
3.12 Grouping and Joining Operators- PART-213:40
3.13 Sorting Operator6:50
3.14 Combining and Splitting Operators9:30
Explore the union and split operators in Apache Pig. Learn how union merges two relations, while split partitions a relation into several outputs.
3.15 Diagnostic Operators10:58

4.1 Eval Functions PART-117:02
Learn eval functions in pig latin: the average function ignores nulls and computes gpa after grouping by name; also explore count, count star, diff, and concatenation with going got.
4.2 Eval Functions PART-24:13
4.3 Eval Functions PART-318:51
4.4 Load and Store Functions16:52
Explore load and store functions in Pig, including bin storage, JSON loader, JSON store, and big storage, to read and write data with schema in binary, JSON, and text formats.
4.5 Tuple and Bag Functions6:19
Learn to convert multiple fields into a tuple and place them in a bag using the to bag function, with foreach-based transformations and dump inspections.
4.6 String Functions11:38
4.7 Math Function8:48

6.1 File Compression in pig10:18
Learn how to handle file compression in Pig by loading compressed input datasets, processing on the fly, and producing compressed output using proper extensions.
6.2 Intermediate Compression7:28
6.3 Pig Unit Testing6:21
Test pig scripts locally with the big unit testing framework, enabling unit, regression testing, and rapid prototyping. Group by student id, compute maximum scores, and verify outputs against expected result.

Requirements

Basic Understanding of Hadoop
Basic knowledge of Declarative Language such as SQL
Basic knowledge of Java Programming Language
Basic Knowledge of Big Data is required but not mandatory

Description

Pig is a high-level platform for creating MapReduce programs used with Hadoop. The language for this platform is called Pig Latin. In this course we will go through the PIG data flow platform and the language used by PIG tool. The concepts which are covered in this course are:

Writing complex MapReduce transformations using a simple scripting language.

Basics of Big Data, Hadoop and MapReduce Framework.

PIG Data Model and Different type of operators to operate on datasets.

Built-in Functions as well as User Defined Functions for performing a specific task.

Running PIG Script, Unit Testing and Compression.

Many more advance topics such as Embedding PIG in Java, PIG Macros etc.

All the books and PDFs are included, allowing you to follow along with the author throughout the modules in this course.

Who this course is for:

Students having interest in Big Data and Hadoop Field
Database Developers and Administrator
Software developers want to build their career in Big Data field
Data Analysts
Data Scientists and Resesarcher

Process Big Data using Apache PIG

What you'll learn

Explore related topics

Course content

Module-0 Introduction to the Course4 lectures • 6min

Module-1 Basics of Big Data, Hadoop and Pig9 lectures • 21min

Module-2 PIG Latin Language, Architecture and Modes of Pig5 lectures • 23min

Module-3 Data Model, Operators and Streaming in PIG15 lectures • 2hr 16min

Module-4 Different Kinds of Built-In Functions in PIG7 lectures • 1hr 24min

Module-5 Advanced Pig Latin with Pig Scripts,UDF's,Utility Commands2 lectures • 19min

Module-6 File Compression3 lectures • 24min

Module-7 Advanced Pig Latin4 lectures • 28min

Requirements

Description

Who this course is for: