Big Data Hadoop Developer Course with Handson

Explore the Java-based Hadoop distributed file system (HDFS), a fault-tolerant, high-throughput store that distributes data across nodes with replication, optimized for large streaming data and files.

Learn how the Hadoop distributed file system uses 64 MB blocks and threefold replication across racks to store large data reliably.

Understand hdfs architecture: the name node manages the namespace and blocks, data nodes store blocks, and the secondary name node handles logs and checkpoints, enabling replication and cluster health checks.

Explore end-to-end HDFS file read and write, from client file creation and name node checks to data block replication across data nodes, with DFS output and input streams.

Learn essential HDFS commands for managing the distributed file system, including mkdir, -ls, copy from local, put, cp, chmod, cat, mv, du, and rm -r.

Explore map reduce programming model for distributed, parallel processing of large datasets with map and reduce phases, data locality, fault tolerance, and flexible language support (Java, Python, C++).

Process real-time data with map reduce by splitting input into chunks, converting splits into key-value pairs with a value of one, shuffling by key, and reducing to final sums.

Explore a map-reduce word count example that tokenizes text, distributes data into splits, maps words to keys, shuffles to group by key, and reduces to final word counts.

Explore how HDFS input splits, as the logical unit of work for a map task, differ from physical blocks and how default block size shapes map task counts.

Explain the MapReduce architecture, with a single job tracker coordinating map and reduce tasks, heartbeat every three seconds from task trackers, and input splits processed on data-local task trackers.

Explore how combiners reduce network traffic by processing map outputs before the reduce phase, and how partitioners use a hash function to group keys for reducers.

Explore MapReduce data flow diagrams showing input splits into three parts, separate map phases, sorting, and how merged outputs are transferred to reducers to produce the final output.

Explore a practical map reduce word count program in Java, featuring map, reducer, and driver classes, string tokenizer, and eclipse-based execution to count word occurrences.

Explore advanced MapReduce concepts, including mapper and reducer execution, driver class orchestration, and the differences between the old and new Hadoop APIs, job control, and iterable value passing.

Explore the generic options parser and tool runner in big data hadoop, and learn how to configure map reduce jobs via command line properties, file system settings, and resource management.

Explore serialization and deserialization in Java, using wrapper classes for primitive types, and writing and reading byte streams to transport data across networks.

Learn to chain jobs by defining dependencies and letting job training orchestrate their execution in the correct order. List running jobs and terminate a problematic job by its id.

Explore how the Hadoop distributed cache speeds jobs by caching and distributing required files to task nodes, reducing network bandwidth usage. Specify files with the -files option for distribution.

Explore built-in and custom counters to collect job statistics, such as rows read, bytes written, and memory usage, including enum-based and dynamic counters for debugging and tracking invalid records.

Explore G-Unit regression testing in Java, which accelerates development and improves quality by validating predefined input-output, and compare it to MRI unit testing for mass-produced, small Java jobs.

Explore how Hadoop schedulers allocate resources to running jobs using fair share and capacity strategies, with queues, map and reduce slots, and job trackers for balanced, responsive execution.

Learn data compression in Hadoop to save storage and speed data movement by automatically decompressing input, while choosing codecs for your workload.

Explain how MapReduce processes input and output formats, including text input format and sequence files, with key-value pairs, offsets as keys, and compression and headers.

Explore chain mapping in Hadoop by connecting two mappers: gene mapper one and uppercase mapper, and a word count reducer across two sequential jobs.

Learn how to apply gzip-based compression to map and reduce outputs, implement compression codecs, and verify maximum temperature results from a temperature dataset within a hadoop workflow.

Explore how a distributed cache speeds data access by matching input data against cached words. Implement map-based processing that loads cache files, reads an input file, and counts word occurrences.

Learn how to process forest fire data with mappers and counters in a Hadoop environment to compute monthly maximum wind speed and area burned using custom and dynamic combos.

Learn to build and test a MapReduce word count using MRUnit, including mapper, reducer, and driver tests in a practical Hadoop unit testing session.

Learn how to process multiple inputs in a Hadoop MapReduce task by mapping two text files to extract the first and last values and produce a combined output.

Learn to create and read sequence files from text inputs, converting to sequence format and applying compression. Understand handling input and output arguments and reading by index.

Learn how Apache Pig provides a high level language for expressing data analysis on Hadoop, translating Pig Latin scripts into map-reduce tasks for scalable extract, transform, and load.

Explore Apache Pig Latin, an engine for big data processing with load, stream, filter, join, and sort operators. Run Pig in Grunt shell or batch mode and use user-defined functions.

Learn to run Pig in local and MapReduce modes, comparing single-machine testing with cluster execution on the Hadoop file system, including pseudo-distributed and fully distributed setups.

Parse scripts to check syntax and types, generating a graph of statements and operators; the optimizer applies projection pushdown, and the execution engine runs compiled jobs on Hadoop.

Explore the grunt shell to interact with DFS, mastering mkdir, ls, copy from local, copy to local, cat, and rm -r through practical commands.

Explore Pig Latin statements and their transformations such as aggregate, join, and sort, translated by Pig into MapReduce for execution on Hadoop DFS, with load, foreach, group, and store options.

Explore the Pig data model by distinguishing scalar types from complex types and how schemas are defined when data is loaded from disk, using describe and dump to reveal schema.

Explore complex data types in big data with Hadoop: doubles, bags, and maps, and learn how they nest data of any type, including other complex types.

Discover how arithmetic operators perform calculations in big data, with addition, subtraction, multiplication, division, and remainder, and see their use in generating fields and evaluating conditions within a relation.

Explore comparison operators such as equal, not equal, less than, greater than, and pattern matching, with hands-on filtering of a relation using dump, load, copy, and filter commands.

Explore cast operator concepts in big data processing, comparing implicit automatic type conversion with explicit conversions, and see practical examples of casting integers, bytes, and floats in a Hadoop workflow.

Demonstrate type construction operators, using double construct, back construct, and map constructors to create bags and maps from records with fields name, age, and GPA.

Understand relational operators in programming, how they compare two entities and support operations like loading, storing, filtering, sorting, mining, and splitting, with diagnostic insights.

Load data from the file system with the load operator, specify the ddm, and store results to an output directory, then inspect contents and leverage user-defined functions for processing.

Master filtering operators in big data with Hadoop by applying equals conditions to select subsets, compare values across fields, and refine query results.

Demonstrate cross product, grouping by age, and inner and outer joins (left, right, full) on sample relations A and B, showing how records are matched and grouped.

Explore how to load data into the DFS, and perform inner, left, right, and full joins using a join operator on specified fields, including handling delimiters and schemas.

Explore combining and splitting operators in big data: use union to merge relations A and B, then split a relation into multiple outputs based on conditions.

Apply the limit operator to display three rows, then sort by the third field in descending order, noting that equal keys yield non-deterministic output.

Explore diagnostic operators in big data with describe to view schemas, filter with matches, and group by name, then dump results and examine explain and illustrate execution plans.

Explore big streaming with Pig using Python to compute the cube of numbers from a dataset by streaming data to a Python script and dumping results.

Hive enables managing large datasets in distributed storage, offering indexing, various storage formats, compressed data processing with snappy, built-in functions, and a sql-like query language.

Explore how Hive use case at Twitter handles massive real-time tweet volumes with partitioned and bucketed Hive tables, enabling scalable storage beyond traditional RDBMS.

Hive enables easier big data processing than MapReduce by reducing code, shortening development time, and enabling command-based workflows. It simplifies joins, supports quick queries, and reduces maintenance.

Explore Hive, an Apache data warehouse on Hadoop that enables querying structured data with a SQL-like language and uses an XML-described plan, with support for custom uploads.

Explore the advantages of HiveQL for querying in Hadoop, enabling structured data projections, joins, partitions, and batch processing with MapReduce, plus integration with Apache DFS and Apache HBase.

Explore Hive architecture, including the shell, metastore, driver, compiler, and execution engine, and how a directed acyclic graph plan guides map-reduce style execution with intermediate outputs.

Explore Hive data types, including primitive and complex types such as maps, struct, and unions, with emphasis on integers, booleans, strings, timestamps, dates, and null values.

Learn Hive query language for processing structured data, using data definition language commands to create, alter, and drop databases, tables, views, functions, and indexes, with notes on transactions and extensions.

Learn how hive data definition language handles creating and using databases, describing and viewing databases, and managing properties, extended properties, and drop or alter operations, including cascade.

Learn how to create and manage tables in Hive using ddl, including create table if not exists, copying schemas with like, and describing table structures with describe and describe formatted.

Explore hive table types—managed, external, partitioned, and external partition—and learn to create them, assign locations, and partition data by country and state.

Learn advanced ddl commands on tables, including drop if exists, rename, add partitions, and modify columns. Apply storage properties, file formats, and online/offline partition controls to manage partitioned tables.

Explore Hive file formats and how data encoding determines storage. See how to specify text, sequence file, or RCFile formats when creating tables, as shown with an employee table example.

Learn how to load data into Hive tables using DML, including inserting, retrieving, deleting, and updating records, from local and DFS paths, with partitioned tables and required partition values.

Learn to manage query outputs in Hive by inserting into other tables, creating partitioned tables with dynamic and static partitions, CTAS, and exporting data to local directories.

Explain HiveQL queries, distinguishing select and action queries, and demonstrate using the select from clause with aliases to fetch names and salaries from employees, including map fields.

Explore Hive operators and functions, applying arithmetic and predicate operators to integer columns, and using string, date, and mathematical and aggregate functions on sample data with hands-on examples.

Master Hive clauses to filter, sort, alias columns, and compute data using limit, where, and like. Use nested selects, group by, having, and order by for aggregations on employee data.

Explore hbase, an open source distributed datastore for hosting large tables on commodity hardware, offering random read and write access and fast lookups with replicated storage on dfs.

Trace the history of HBase from Google's Bigtable paper (2006) and its 2007 start as a subproject of Hadoop to an Apache project with 1.0.0 in 2015.

Explore the building blocks of HBase: tables, rows via row keys, and column families with qualifiers; see how cells combine these elements and how regions split automatically.

Explore how column families in HBase group data, with each family containing columns and versions stored in an HFile, and how column qualifiers organize data within records.

This lecture explains how column families are stored on disk, with two families for demographic and personal data, including employee name, address, date of birth, and gender as columns.

Explore the HBase data model, where data is stored as key-value pairs with row keys, column families, and qualifiers. Learn how column families can host many columns and be sparse.

Explore timestamp as a cell versioning mechanism, with default write-time timestamps, latest reads, and retrieving prior values by timestamp or null when absent, with three versions kept per column family.

Explore getting started with the HBase shell, learn to communicate with HBase and HDFS, and use commands like status, version, help, and who am I to manage regions.

Explore DDL in HBase, define database structures by creating tables with column families, and manage them with describe, list, exist, disable, and drop commands.

Learn how to use disable and enable commands in HBase, verify status with is disabled and is enabled, and apply bulk changes with a regular expression.

Learn how to use DDL in HBase to modify column families, changing max versions and max file size, delete or add column families, and verify with describe.

Explore DML commands in HBase, including put for inserting cells, get for reading rows, and delete, delete all, and scan to verify changes and truncating to clear data.