Learn Big Data Hadoop: Hands-On for Beginner

Learn how Hadoop enables distributed storage and parallel processing of massive data using commodity hardware, with components like HDFS, YARN, and MapReduce, plus data replication for fault tolerance.

Understand how Hadoop, an open source distributed framework on commodity hardware, enables scalable processing of structured and unstructured data. See use cases from Facebook, Rackspace, and SpaceX.

Explore the Hadoop ecosystem, from HDFS and MapReduce to scalable processing across thousands of nodes, with Sqoop for data transfer, Mahout for machine learning, and ZooKeeper for coordination.

Compare structured data in spreadsheets with unstructured data like video and social media to illustrate analysis challenges. Highlight semi-structured data as a middle ground for ai-driven business insights.

Explore the relationship between big data and Apache Hadoop, an open source framework, which processes and stores large, diverse datasets using commodity hardware and the Hadoop distributed file system.

Explore the future of big data and Hadoop, including rising data to 463 exabytes per day by 2025, and cross-industry adoption across banking, healthcare, education, and government.

Identify challenges with big data, including the shortage of skilled data professionals—data scientist, data analyst, and data engineer—and rapid tool evolution, while addressing data integration, unstructured data growth, and security.

Compare Hadoop, an open source framework for storing and processing big data, with IBM's RDBMS for structured data and OLTP; Hadoop handles structured, semi-structured, and unstructured data at petabyte scale.

Differentiate Hadoop from data warehouses by comparing data types, schema-on-read versus schema-on-write, and cost, noting Hadoop's efficiency on commodity hardware and use in data science and data engineering.

Explore the differences between hadoop and teradata, contrasting teradata's scalable relational data warehouse with hadoop's open source big data framework and master-slave versus massive parallel processing architectures.

Explore the two types of big data projects: on premises and cloud-based, and compare full control of on-premises infrastructure with cloud scalability and cost efficiency.

Explore the cluster environment in Hadoop, where multiple server instances span nodes with identical configurations, each node performing a different task to ensure high availability and scalability.

Understand how a Hadoop cluster, a Java-based open source framework, uses master and slave nodes to run parallel tasks with MapReduce for distributed big data analytics on low-cost hardware.

Start a single-node Apache Hadoop 3.3.0 cluster, monitor resource manager logs for startup status, and inspect the DFS filing system details.

Learn to stop the Apache Hadoop 3.3.0 single-node cluster, manage Java processes, and observe system status to ensure the Hadoop services shut down cleanly.

Learn how to start a single-node Apache Hadoop cluster on Ubuntu, initialize HDFS and YARN, and monitor jobs via web UIs.

Stop the Apache Hadoop 3.3.0 single-node cluster on Ubuntu by halting the distributed file system, then stopping data nodes and the secondary namenode, confirming all subsystems are shut down.

Explore the Hadoop distributed file system (HDFS), designed for commodity hardware with fault tolerance and high availability, using triple replication for durability and high throughput for large data.

Explore the Hadoop filesystem (fs) shell, learn to interact with local and Windows-based file systems, and practice specifying host, port, and file names for cross-platform access.

Learn to use the Hadoop filesystem shell command to check the Hadoop version in a hands-on, beginner-friendly lesson, with reference to the course materials and source code repository.

Explore the file system shell commands and their options, and learn how to get help for any command. View the list of commands and options and how to use them.

Explore hands-on file system shell commands to create directories in hdfs, including creating subdirectories, practicing directory creation and basic navigation in a Hadoop environment.

Learn how to use the cat command in the filesystem shell to display data from a file location, view appended files, and use head for large files.

Explore common filesystem checks using shell commands and checksum verification in a hands-on practical demonstration, revealing how to locate information and assess file data efficiently.

Explore how to use the FileSystem shell command copyFromLocal to move files from the local environment to the distributed file system, with workflows on Windows or Linux.

Engage in a hands-on demonstration of the file system shell command copyToLocal within a local environment, copying files to a local destination and validating results.

Explore filesystem shell commands to count and locate parts of a file, using options that control output columns. The lecture demonstrates a practical iPhone-based demo of these commands.

This lecture demonstrates copying files with the filesystem cp command, moving data from a source directory to a destination directory, with options -i (prompt) and -B (show attributes).

Work through using the df command to display free space, perform a practical demonstration, and verify disk space availability for a Hadoop file system.

Learn to use the du command to display file and directory sizes, with options like -h for human-readable output and -s for aggregate summaries.

Learn to use the filesystem shell find command to search for files by name, print results to standard output, and apply modifiers for practical, scalable file discovery.

Engage in a hands-on session on the filesystem shell, learning how to locate and retrieve files using the get command and execute shell commands in a practical Hadoop practice.

Participate in this hands-on demo to use the getfacl shell command to inspect file access control lists, view user, group, and permission attributes, and troubleshoot access issues.

Master the head command in the Hadoop file system shell to display the first lines of output on the monitor. Follow practical steps to limit results and view results clearly.

Explore the filesystem shell command ls in a hands-on, practical demonstration, learning how to use options and arguments to list and display items.

Demonstrate using the file system shell command moveFromLocal to transfer a file from the local filesystem into the distributed storage, deleting the source after the move.

Master the mv command in the Hadoop distributed file system to move files within directories, demonstrated by directory-to-directory moves, with cross-file-system moves not permitted.

Learn how to use the filesystem shell put command, specify destination and local paths, manage existing files with flags, and follow a practical demonstration.

Engage in a hands-on exploration of the FileSystem shell rm command within the big data hadoop course, demonstrating practical rm options and file deletion workflows for beginners.

Demonstrates using the filesystem shell rmdir to remove a subdirectory, with a practical walkthrough of deleting a directory and handling non-existent paths.

Practice using the filesystem shell command tail to display the last kilobyte of a file, with a practical demonstration of command options.

Practice using the Hadoop file system shell command touchz to create a zero-length file and trigger the next step in the process.

Learn how to use Hadoop fs -appendToFile to append local files to the Hadoop file system, including single or multiple sources and input from standard input, with practical demonstrations.

practice using the chgrp command to change a file's group ownership, guided by a practical demonstration of permissions, ownership, and group changes on a sample path.

Learn to use the filesystem shell chmod command to change file permissions in Hadoop, and explore owner, group, and others settings with examples like 777 and 744.

Learn how to use the chown command in the file system shell to change a file's owner, with a practical demonstration showing owner verification and required privileges.

Explore filesystem shell command getmerge in a Hadoop environment to merge files from a source directory into a local destination file, and copy results from Hadoop to the local system.

Demonstrates the filesystem setrep command to change the replication factor from the default three, including recursive changes for directories, with a practical example on /user/data engineer/files.

Learn the filesystem command stat to view file statistics such as permissions, size, owner, group, and timestamps.

Learn how the touch command updates a file’s access and modification times in the filesystem shell within the Hadoop environment, creating a zero-length file if it does not exist.

Learn how to use the Hadoop filesystem concat command to merge two source files into a single target file in the same directory, with a hands-on demonstration.

Explore how to use the filesystem shell to display the classpath in Hadoop, locating the Hadoop jar and required libraries through practical command demonstrations.

Explore five system commands for environmental variables and demonstrate displaying Hadoop environmental variables through a practical demonstration, including querying and reading the declared environment details.

Demonstrates using HDFS dfs fsck to check health and consistency, report missing or under-replicated blocks, and view file-, block-, and location-level details across a Hadoop setup.

Explore filesystem shell getconf usage to retrieve dfs configuration details for name node, secondary name node, and backup nodes, with a practical demonstration of the getconf utility.

Explore file system command groups and how to view group information for a single user, performing practical steps with DFS groups for the data engineer group to troubleshoot file accessibility.

Explore filesystem shell commands for Hadoop data nodes, view and troubleshoot data processes by listing process IDs, and learn admin steps like DFS data load restart in a practical demonstration.

Explore the HDFS overview, highlighting fault tolerance, high throughput, streaming data access, and the write-once read-many model for large datasets on commodity hardware.

Explore the hdfs architecture, including a single name node managing the filesystem namespace, data blocks stored on data nodes, and commodity hardware running java-based dfs software.

Explore how HDFS storage uses data blocks to store large files with write-once, read-many semantics and streaming efficiency. Configure block size in dfs, for example 134217728 bytes (128 mb).

Explore Hadoop installation modes—local standard mode, pseudo distributed mode, and fully distributed mode—and their use for development and testing on a single machine, as well as staging and production environments.

Understand that the name node is the centerpiece of HDFS, stores directory tree metadata and tracks file locations across the cluster, while data nodes do not store the actual data.

Explore how data nodes serve read and write requests, manage block creation and replication, and store blocks on disk while sending heartbeats to the NameNode to confirm uptime.

Learn how the node manager launches containers on a node, executes app master tasks, and runs disk and task checks that mark unhealthy nodes via heartbeat to the resource manager.

Explore how the resource manager tracks cluster resources and schedules applications, including MapReduce jobs, serving as the central authority for resource allocation and job scheduling.

Explore how the secondary name node merges the image and edit logs at startup to keep the edit log size in check, reducing restart delays on busy clusters.

Explore data replication in a large Hadoop cluster, where blocks are replicated for fault tolerance. Learn how replication factor and variable block size enable flexible, reliable storage.

Learn how rack awareness guides block placement during read and write operations in a Hadoop cluster to improve fault tolerance and data reliability.

Explore robustness in hadoop's pfs by ensuring data reliability through heartbeat, replication, checksums, and snapshots amid node failures and partitions.

Explore how hdfs snapshots capture a point-in-time state to protect against user error and disaster recovery, with modifications tracked in reverse chronological order and memory used only when changes occur.

Explore how DFS balances block placement across data nodes to accommodate new nodes and replicas. Learn balancer tool and service modes, balancing conditions, and configuring intervals for rebalancing.

Explore Yarn, the redesigned resource manager in Hadoop, and how it allocates cluster resources and schedules tasks. It enables graph, interactive, stream, and batch processing on a scalable, multi-tenant platform.

Explore the difference between MapReduce and YARN, focusing on MapReduce's data processing on a cluster. MapReduce previously managed data processing and resource management; YARN adds separate resource management.

Explore yarn architecture by separating resource management from job scheduling into a global resource manager and an application master, including the scheduler, reservations, and federation across clusters.

Explore how capacity scheduler and fair scheduler allocate resources in the Hadoop resource manager, ensuring fair sharing, minimum capacity guarantees, and elastic access to excess capacity across organizations.

Learn to run mapreduce on yarn with Hadoop 3.3 using predefined mapreduce examples, submit 16 jobs from Neon, and view the 3.142 output.

Explore the YARN web UI to monitor MapReduce jobs, view nodes and applications, access logs, and troubleshoot via system logs, errors, and resource manager and node manager details.

MapReduce splits large datasets into independent chunks, processes them in parallel with map and reduce tasks, and coordinates via the Hadoop distributed file system and a scheduler.

Explore how MapReduce distributes code across the cluster, including Hadoop clusters, to run where the data resides, enabling scalable processing of structured and unstructured data.

Explore MapReduce limitations and why Spark's in-memory processing gains favor, noting disk-based mapper output slows I/O, and post-map reduce execution.

Maps input records into intermediate key-value pairs, possibly different from the input type. Configures one mapper per input split and enables a combiner before reducers.

Learn how a reducer aggregates intermediate values by grouping keys, follows shuffle, sort, and reduce phases, and writes each key with its values via the framework, with configurable reducer count.

Explore shuffle in MapReduce, which guarantees that every reducer receives input sorted by key during processing.

Explore how salt affects the map output and how the sort and shuffle phases order input, with the sort step listing input in sorted order.

Explore secondary sort by configuring a comparator class to control how intermediate keys are grouped, differing from pre-reduction grouping rules, and apply it to sort values in the Hadoop job.

Determine the map count from input size and block count, targeting about 10–200 maps per node (up to 300 for simple tasks) so maps run at least a minute.

Explore how to determine the right number of reduces in Hadoop by balancing map output with load balancing factors like 0.95 and 1.75, improving performance and resilience.

Learn to configure a Hadoop job with no reducer by setting reduce tasks to zero, sending map outputs to file system via the file output format to an output path.

Understand how partition selects which keys from intermediate map output reach reduce tasks via a hash function, with partitions equal to the number of reduce tasks and partition as default.

Learn how the counter facility in MapReduce reports statistics, using the statistics member, while the read user implementation can use the counter to report statistics with bundled mapper and reducer.

Explore inputs and input splits, where a split represents data processed by a single mapper, as a byte-oriented and record-oriented view by the record reader.

Explore how record readers convert byte oriented input from an input split into key-value records for the mapper, present keys and values, and manage record boundaries.

Explore a MapReduce example that uses four mappers to tokenize the poem into words, assigns one count per word, and a reducer to aggregate word counts.

Learn to unzip gz files stored in HDFS by piping Hadoop fs-cat to gzip -d and Hadoop fs-put, handling single and multiple files with practical command guidance.

Unzip gz files into a new directory in Hadoop using Hadoopfs-put, gzip-d, and Hadoopfs-cat to move, unzip, and verify content in HDFS.

Identify disk IO as the bottleneck in HDFS put for large files; block size changes don't boost throughput; boost ingestion with parallel uploads or ingestion tools like Flume or Kafka.

Explore how Hadoop NameNode high availability works, with active and standby nodes, journal nodes, and edit logs, ensuring quick failover and split-brain prevention via ZooKeeper failover controller.

Learn that Hadoop MapReduce jobs do not have to be written in Java; implement them in languages such as Ruby, Perl, Python, or R via the Hadoop streaming API.

Explore Apache Pig, a high-level platform for expressing data analysis program on Hadoop. Write Pig Latin to run on MapReduce or Apache Spark, and extend with user defined functions.

Compare MapReduce and Apache Pig, where MapReduce is a processing paradigm and Pig is a high-level data flow language. Note that MapReduce requires more code, while Pig delivers concise code.

Explore Apache execution modes—local mode and MapReduce mode—through a hands-on demo using a local file, running Hadoop, and submitting MapReduce jobs to display results.

Master batch mode with a big script that consolidates pig script and pig latin commands for local and MapReduce runs, and view batch job results.

Learn how pig latin statements transform input relations into output relations, with expressions and schemas, ending in a semicolon, and align the load, transformations, and store steps.

Explore simple data types such as integer, long, float, double, character, and string, and examine complex datatypes like bag and collection of key-value pairs in Apache.

Explore simple data types by creating a sample student file with numbers, floats, doubles, and characters, then load it in local mode using a comma delimiter and print the content.

Demonstrate a complex datatype in Hadoop using a map to store department id, name, and a map address with city and state; show loading with quick storage and delimiter options.

Explore loading data from the file system using the load function and various storage options (big storage, json loader, json storage, binary storage), with schema definitions for columns.

Explore data transformation with core operators in Hadoop: filter, for each, group, inner and outer joins, union, and split, through practical, hands-on demonstrations.

Learn how to use the filter operator to select rows by a boolean expression on columns a1, a2, and a3, and see the filtered results via a MapReduce job.

Demonstrate the foreach operator to generate data transformations by selecting columns, applying generate expressions, and defining aliases and schemas through a hands-on file example.

Apply the group operator by loading a sample student data file, describing the schema, and grouping records by age to reveal the resulting age-based groups.

This hands-on lecture demonstrates the co group operator in Hadoop, loading two relations, grouping by owner and frame, and displaying the resulting grouped output.

perform a hands-on join operation on two data files by aligning records on common keys, loading them from storage, and validating a seven-record result.

Perform a hands-on union of two datasets to demonstrate the union operator. Load a three-column two-row file and a two-column three-row file, yielding five records.

Learn to use the split operator in a hands-on Hadoop demo to partition a three-column, integer dataset, load a file, and apply a split condition to filter records.

Store data into the file system using the default storage function, with the using clause and storage delimiters, and verify the output in the generated my output directory.

Debug pig latin with four operators: build a process operator to display terminal results, describe to review schema, explain to view logical, physical, and MapReduce plans, illustrate step-by-step execution.

Explore the dump operator by loading a file, defining a name and ages schema, exploding data, and filtering results by name for display.

This hands-on lecture demonstrates the describe operator, which returns the schema of a relation using an alias, with a practical example on a student file.

Explore the explain operator to view its execution plans, including logical, physical, and MapReduce plans, plus memory usage.

Demonstrate the illustrate operator with a step-by-step execution of a sequence of statements, including loading a file, grouping by name, and counting occurrences to reveal the output structure and result.

Apply comparison operators to filter data using equal to, less than, greater than, less than or equal to, greater than or equal to, and matches, on filter and student files.

Explore the order by operator to sort a relation by one or more fields, with ascending or descending options, and see a practical demonstration using a data file.

Learn how the rank operator assigns a rank to each record in a relation, using ascending and descending orders with order by clause, demonstrated by loading data and serial numbers.