Big Data Internship Program - Data Ingestion-Sqoop and Flume
4.2 (63 ratings)
Instead of using a simple lifetime average, Udemy calculates a course's star rating by considering a number of different factors such as the number of ratings, the age of ratings, and the likelihood of fraudulent ratings.
918 students enrolled
Wishlisted Wishlist

Please confirm that you want to add Big Data Internship Program - Data Ingestion-Sqoop and Flume to your Wishlist.

Add to Wishlist

Big Data Internship Program - Data Ingestion-Sqoop and Flume

Complete Reference for Apache Sqoop and Flume
4.2 (63 ratings)
Instead of using a simple lifetime average, Udemy calculates a course's star rating by considering a number of different factors such as the number of ratings, the age of ratings, and the likelihood of fraudulent ratings.
918 students enrolled
Created by Big Data Trunk
Last updated 11/2016
English
Current price: $10 Original price: $195 Discount: 95% off
5 hours left at this price!
30-Day Money-Back Guarantee
Includes:
  • 2.5 hours on-demand video
  • 9 Supplemental Resources
  • Full lifetime access
  • Access on mobile and TV
  • Certificate of Completion
What Will I Learn?
  • After this course, students will have knowledge and understanding of Data Ingestion .
  • Have excellent understanding of Apache Sqoop and flume tool with hands-on experience .
  • Understand the working of a project in real-world scenario.
View Curriculum
Requirements
  • Should know the basics of BigData concepts like-HDFS,MapReduce and some knowledge of RDBMS.
  • Should take our Part-1 free course to understand these concepts better. (Not mandatory but desirable).
Description

This course is a part of “Big data Internship Program”  which is aligned to a typical Big data project life cycle stage.

  • Foundation
  • Ingestion
  • Storage
  • Processing
  • Visualization

This course is focused on the Ingestion in Big data . 

Our Course is divided into two part 1) Technical Knowledge with examples and 2) Work on project 

Technical Knowledge 

  1. Big Data ingestion's concept and means 
  2. Sqoop concept and feature.  
  3. Good understanding of sqoop tools with arguments 
  4. Flume concept and configuration 
  5. Flume features: Multiplexing,Flume Agents,Interceptors etc .
  6. Understanding of different File Format supported by Hadoop 

Project Part 

  1. Get the access to our private GitHub repository 
  2. Build the first part of Our Recommendation Book project using sqoop and flume
Who is the target audience?
  • This course is for anyone who wants to learn about Data Ingestion in Hadoop Ecosystem with Sqoop and Flume.
  • Students who want to do internship.
  • Big Data Analytics Professional.
Students Who Viewed This Course Also Viewed
Curriculum For This Course
29 Lectures
02:20:14
+
Introduction
4 Lectures 20:11

In this video, we have explained what is data ingestion, How to process data, challenges in data ingestion, the key function of data ingestion.

Preview 05:53

This part -1 course is focused on the foundation of Big data . It covers technical  items like

Technical Foundation

  • Refresh your knowledge on Unix
  • Java based on usage into Big Data .
  • Understand  git /github which is used by most of the companies for source control
  • Hadoop Installation

Part - 1 is free here 

https://www.udemy.com/big-data-internship-program-part-1-foundation


Recap - Big data Internship Program - Part 1 Foundation
03:52

In this video, we have explained what is data ingestion and there tools available in markets.

Data Ingestion Tools
04:48

In this video, we have explained data ingestion tools Kafka, Chukwa,Storm etc.

Some more Data Ingestion Tools
05:38
+
Different types of File Formats in Hadoop
6 Lectures 22:57

This video shows different type of file format supported in Hadoop.

Introduction to FileFormats
04:04

Introduction to File Formats
2 questions

CSV /Text files are quite common and often used for exchanging data between Hadoop and external systems.

Text/CSV file formats
03:01

Text/CSV file formats
2 questions

This video shows that Sequence files store data in a binary format with a similar structure to CSV. Like CSV, sequence files do not store metadata with the data so the only schema evolution option is appending new fields. 

BinaryFileFormats-Sequence Files
02:47

BinaryFileFormats-Sequence Files
2 questions

Avro files are quickly becoming the best multi-purpose storage format within Hadoop. Avro files store metadata with the data but also allow specification of an independent schema for reading the file. Here we show you all about this file format .

BinaryFileFormats-Avro
04:24

BinaryFileFormats-Avro
7 questions

RC Files or Record Columnar Files were the first columnar file format adopted in Hadoop. Like columnar databases, the RC file enjoys significant compression and query performance benefits.ORC Files or Optimized RC Files were invented to optimize performance in Hive and are primarily backed by HortonWorks. This video shows about these two file format.

Columnar formats-RC and ORC files
04:31

Columnar formats-RC and ORC files
5 questions

Parquet Files are yet another columnar file format that originated from Hadoop creator Doug Cutting’s Trevni project. Like RC and ORC, Parquet enjoys compression and query performance benefits, and is generally slower to write than non-columnar file formats.  In this video you can learn more about this file format .

Columnar format-Parquet Files
04:10

Columnar format-Parquet Files
3 questions
+
Sqoop
8 Lectures 45:12

In this video, we have explained to you what is sqoop, what is flume,  sqoop work flow, sqoop architecture.

Introduction to sqoop
05:53

Introduction to Sqoop
3 questions

In this video, we have explained what is import command, how sqoop import command is executed.

Sqoop Import
04:23

Sqoop Import
1 question

In this video we have explained how to execute commands in terminal,how to get table list, how to get list of data bases, how to import data in hdfs.

Import data from MySql to HDFS
06:38

Import data from MySql to HDFS
3 questions

In this video, we have explained how to run sqoop commands, what is structure of sqoop commands, what are the parameters used in the execution of sqoop commands.

Other variations of Sqoop Import Command
05:32

Other variations of Sqoop Import Command
5 questions

In this video we have explained what is sqoop export, and how it is used.

Running a Sqoop Export Command
05:50

Running a Sqoop Export Command
2 questions

In this video, we have explained what is sqoop jobs how it used and when it is used. how to create jobs, how to list sqoop jobs available.

Sqoop Jobs
05:41

Sqoop Jobs
3 questions

In this video we have explained what is incremental sqoop, and how it works.what are the incremental import parameters etc.

Preview 05:46

Sqoop incremental import
3 questions

In this video, we have explained how incremental import works, how to append data to the table.

Lab: Sqoop incremental Import
05:29

Test Your Sqoop Knowledge
20 questions
+
Flume
8 Lectures 30:45

In this video, we have explained what is flume, and where it is used.difference between flume and sqoop.

What is Flume?
02:31

What is Flume?
2 questions

In this video, we have explained how flume works, what is flume agent what are the components of flume agent, how data is flow between various components of the flume.

Data Flow Model
04:24

Data Flow Model
2 questions

In this video, we have explained what are components of the flume, how they are configured i.e how flume agent is configured.

Flume Configuration File
04:25

Flume Configuration File
3 questions

In this video, we have explained how to run flume agent. and get a result.

HelloWorld example in Flume
06:09

In this video, we have explained what is multi-agent flume, what is the consolidation of flume.

Multi Agent flow
02:37

Multi Agent flow
2 questions

In this video, we have explained what is multiplexing,use of multiplexing, channel selector etc.

Multiplexing
05:05

Multiplexing
2 questions

In this video, we have tried to explain what is an interceptor, why it is used, how it is configured, and how this runs. what are types of interceptors?

Interceptors in Flume
02:30

Interceptors in Flume
1 question

Test Flume Knowledge
10 questions

In this video, we have tried to explain what is Recommendation with the help of book recommendation concepts.

Preview 03:04

Book recommendation Project Overview
3 questions
+
Project Work
3 Lectures 21:09

In this video, we have shown you how to load data in MySQL and then how to import data in hdfs. through  sqoop commands.

Book recommendation Project Sqoop Work Part-1
11:27

In this video, we have explained what is a script,how we can execute our job by using the shell script. 

BookReccomendation Project- Sqoop Work -Part2
02:37

In  Video, we have shown how book recommendation is working, how the rating is generated in hdfs through the flume.

Book recommendation Project - Flume Work
07:05
About the Instructor
Big Data Trunk
4.1 Average rating
752 Reviews
13,020 Students
4 Courses
All about Big Data and Hadoop

Big Data Trunk is the leading Big Data focus consulting and training firm founded by industry veterans in data domain. It helps is customer gain competitive advantage from open source, big data, cloud and advanced analytics. It provides services like Strategy Consulting, Advisory Consulting and high quality classroom individual and corporate training.