Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Learn How to Create Hadoop MapReduce Jobs in Python
Rating: 3.4 out of 5(37 ratings)
658 students

Learn How to Create Hadoop MapReduce Jobs in Python

Hadoop MapReduce Jobs Using Python
Created byInflame Tech
Last updated 8/2020
English
English

What you'll learn

  • Understand what is Hadoop?
  • Understand MapReduce i.e. Heart of Big Data and Hadoop.
  • Running Jobs using Python.
  • Design and Implement Mapper and Reducer phase in Python
  • Execute and Run Hadoop Streaming Jobs
  • Integrate Mapper phase and Reducer phase with Java Driver Class

Course content

1 section • 37 lectures • 4h 40m total length

  • 1.1 prerequisites0:44
  • 1.2 Course Module2:30
  • 1.3 Why MapReduce with Python1:45

    Explore why MapReduce with Python offers a flexible, cost-effective platform for big data projects. Learn to write MapReduce programs in Python, avoiding Java onboarding and speeding development.

  • 2.1 What is Apache Hadoop7:47
  • 2.2 Comparison with RDBMS4:10
  • 2.3 HDFS in Hadoop7:29
  • 2.4 Cluster modes of Hadoop2:42

    Explore Hadoop cluster modes (standalone, pseudo distributed, and fully distributed), covering local file system input/output, debugging use, single-machine daemons, and production-scale multi-node data.

  • 2.5 HDFS and MapReduce4:47

    Learn how Hadoop uses HDFS to store big data and Apache MapReduce to process it across a cluster, with map and reduce phases transforming data blocks into outputs.

  • 3.1 MapReduce Model3:39
  • 3.2 Why MapReduce5:25

    MapReduce splits data into map and reduce tasks to enable scalable analysis of big data on Hadoop. It hides parallelization details and supports languages like Java and C#.

  • 3.3 Map and Reduce Operation5:19

    Explore the anatomy of a Hadoop mapreduce job, where map tasks process data blocks into key-value pairs and a shuffle and sort stage feeds the reduce phase into final output.

  • 3.4 Data Flow In MapReduce5:35

    Explore data flow in Hadoop MapReduce, detailing how input data is divided into blocks, processed by map and reduce tasks, and shuffled to produce final output stored in HDFS.

  • 3.5 MapReduce Daemons7:07

    Explore how Hadoop MapReduce daemons operate, focusing on the job tracker and task tracker, master and slave nodes, and the lifecycle from input in HDFS to map and reduce tasks.

  • 4.1 Introduction to Hadoop Streaming4:31

    Explore how Hadoop streaming lets you run map and reduce with any executable or script, using standard input and output to build scalable MapReduce jobs with unix-style pipelines.

  • 4.2 Streaming Command Options7:28
  • 4.3 Generic Command Options3:11

    Master the generic command options for Hadoop streaming, including -conf, -D, -fs, and -jt, to configure and run mapreduce jobs in Python with archives and files.

  • 4.4 MapReduce Sample Program-128:56
  • 4.5 MapReduce Sample Program-221:52

    Learn how to create a Python-based Hadoop MapReduce job that analyzes temperature data to determine the maximum temperature per year using map and reduce scripts.

  • 5.1 Chaining of MR Jobs10:34

    Learn how to chain multiple MapReduce jobs into a single workflow using a driver that configures and runs job1 and job2, passing outputs as inputs.

  • 5.2 Custom Combiner17:51

    Learn how to implement a custom combiner to merge intermediate outputs in Hadoop MapReduce jobs, improving efficiency and data flow in Python-based workflows.

  • 5.3 GenericOptionParser3:28

    Learn how to use the generic option parser to handle user options for Hadoop MapReduce jobs in Python, configure runners, and implement the driver to run jobs.

  • 5.4 Distributed Cache11:17

    Learn how to use the distributed cache in Hadoop MapReduce by specifying cache files in the job configuration, then read cached data during map tasks alongside standard input.

  • 6.1 JUnit Testing9:27

    Explore g-unit unit testing for hadoop mapreduce jobs in python, including building a mapper, defining a best_one function, importing modules, and validating input-output with assertions.

  • 6.2 Analysis of IRIS dataset11:11

    Analyze the IRIS dataset with Python in a Hadoop MapReduce context, using a driver class to run MapReduce jobs and manage data with variables and a dictionary.

  • 6.3 Built-in and Custom Counters in Hadoop12:58
  • 6.4 Custom Partititioner5:47
  • 6.5 Hadoop Sequence File Format6:09

    Learn how the Hadoop sequence file format stores key-value pairs in blocks with headers and versioning, using optional compression for efficient MapReduce processing in Python.

  • 6.6 Read Write Sequence File5:27

    Learn how to create Hadoop MapReduce jobs in Python that read and write sequence files, configure input and output formats, and leverage the sequence file format with compression.

  • 7.1 Hadoop Data Types1:46

    Hadoop Data Types

  • 7.2 Processing of XML File6:49

    Processing of XML File

  • 7.3 Data Compression with Hadoop15:14

    Data Compression with Hadoop

  • 7.4 Data Serialization using Avro-Theo9:21

    Data Serialization Using Avro Theo

  • 8.1 Limitations of Hadoop 1.x7:28

    Limitation of Hadoop 1.X

  • 8.2 Hadoop 2.x with YARN7:27

    Hadoop 2.x with YARN

  • 8.3 YARN and its Processing Application4:54

    YARN and its Processing Application

  • 8.4 YARN MR Application Execution Flow5:26

    YARN MapReduce Application Execution Flow

  • 8.5 Hadoop 2.x Cluster Architecture2:56

    Hadoop 2.x Cluster Architecture

Requirements

  • Basics of Computer Science
  • Basics of Hadoop would be benificial but not required
  • Basics of Object Oriented Programming

Description


Apache Hadoop is an open-source software framework for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware. MapReduce is the heart of Apache Hadoop. MapReduce is a framework which allows developers to develop hadoop jobs in different languages. So in this course we'll learn how to create MapReduce Jobs with Python.This course will provide you an in-depth knowledge of concepts and different approaches to analyse datasets using Python Programming. 

This course on MapReduce Jobs with Python will help you to understand MapReduce Jobs Programming in Python, how to set up an environment for the running MapReduce Jobs in Python, how to submit and execute MapReduce applications in Python environment. We will start from beginning and then dive into the advanced concepts of MapReduce.

Who this course is for:

  • Big Data Professionals
  • Hadoop Developers
  • Python Developers who want to go in the field of Big Data
  • Students who are interested in Hadoop MapReduce

Report abuse