Master Apache Hadoop - Infinite Skills Hadoop Training

Understand Hadoop architecture with the two pillars: HDFS storage and MapReduce processing, and see how master and slave daemons coordinate, scale, and self-heal a cluster.

Provide a concise overview of Hadoop software requirements and large-cluster OS setup, endorsing Linux with Cloudera Manager, package installs over tarballs, and validated hardware with NTP.

Explore how EMR on AWS abstracts cluster management, integrates with S3 and EC2, and supports scalable Hadoop workflows with Redshift and Kinesis.

Learn to deploy a four-node hadoop cluster on Amazon Web Services using Cloudera Manager, with a master for HDFS name node and MapReduce job tracking and four slaves.

Learn to install Hadoop with CDH and Cloudera Manager on AWS Ubuntu, including preparing SSH access, updating packages, running installer, configuring Java JDK, and accessing the Cloudera Manager web portal.

Deploy and configure a CDH cluster using Cloudera Manager's web GUI, assign master and slave roles, install CDH packages, input private IPs, and verify all services with green checks.

Set up an Amazon cluster, install Cloudera Manager and CDH, load Shakespeare data, run a word count MapReduce job to validate HDFS, and monitor progress.

Explore hive and pig interfaces, their value propositions, and a side-by-side syntax comparison that shows how each translates queries into mapreduce jobs.

Install and validate the Cloudera QuickStart VM locally, including VMware Fusion/Player, then start Cloudera manager and run a sample Hadoop job to verify the setup.

Explore the HDFS architecture with name node, secondary name node, and data nodes; learn about blocks, replication factor, worm, heartbeats, block reports, and checksums for data integrity.

Walks through the HDFS file write process, showing how the client, name node, and data nodes handle block size and replication factor, block creation, replication, and a write pipeline.

Load and manage data in HDFS with the Hadoop FS command line, creating input directories and loading datasets, while examining HDFS usage and name node web interfaces.

Explore hdfs using fs commands like ls, cat, and tail to inspect input data and datasets, browse the name node web GUI, and understand the trash directory.

Trace the evolution of HDFS access controls from Linux-like permissions to Kerberos authentication, then learn quotas, sandboxing with trash directories, and provisioning techniques.

Explore the mapreduce paradigm and how mappers and reducers enable mass-produced parallel processing, including input splits, the number of mappers and reducers, map-only jobs, and partitioners.

Explore the mapreduce architecture, compare version 1 with version 2, and outline the job tracker, task trackers, map and reduce tasks, data locality, and key configuration files.

Explore the mapreduce word count in java by walking through the driver, jar submission, compilation, and input-output setup, with map and reduce roles.

Examine the map and reducer code line-by-line, then run a word-count MapReduce job on Shakespeare data, compile a jar, and verify the output.

learn when to use java in hadoop, master serialization, and understand ridable and ridable comparable key value types, including text, integers, and longs for mapper and reducer workflows.

Explore hive basics, architecture, and how hive maps data in hdfs via the metastore, including internal vs external tables, megastores, partitions, and buckets, with example queries and limitations.

Explore Hive patterns and anti-patterns, noting Hive's convenience and sql-like language for ad hoc analyses, and compare when to choose Impala or Pig for performance.

Explore pig basics, including architecture and data structures (atoms, data bags, tuples, maps), and learn to load, filter, and for each generate in pig latin on a cluster.

Identify pig’s strengths for convenience in java-free environments and handling complex or semi structured unstructured data, while noting when hive or impala are preferred for analytics and performance.

Explore Hadoop import and export options, comparing Flume, Scoop, Storm, Kafka, and Kinesis. Learn best practices, three-tier directory flows, and strategies to handle small files and end-to-end data workflows.

Compare webhdfs and httpfs for restful hdfs access using curl commands. Fuse dfs mounts hdfs as a local drive, enabling safer access and basic quotas.

Introduces scoop as the rdbms import/export tool, compares scoop and scoop to services, and demonstrates advanced usage and sample commands for importing and exporting data.

Work through a lab that uses Sqoop on a QuickStart VM to import and export data between my sequel, HFS, and Hive, with data validation.

Master Apache Oozie introduces orchestration of Hadoop jobs through workflows with control and action nodes, featuring scoop, hive, pig, and java actions, and supports on-demand or scheduled execution.

Wrap up covers the Hadoop core, architecture, and cluster management, and shows how to write basic MapReduce programs. Explore intermediate Hive, Pig, Flume, and Usie usage to analyze real data.