Hadoop & Spark

Explore why big data requires a dedicated framework like Hadoop to store and analyze massive volumes and unstructured data, highlighting RDBMS limitations in volume, speed, and analytics.

Explore Hadoop intro: compare vertical scale up vs horizontal scale out, master/slave clusters, distributed file system blocks, and replication for fault-tolerant parallel processing.

Explore the Hadoop ecosystem, its HDFS storage, and MapReduce style execution, plus dataflow pipelines, Pig Latin, and streaming concepts with Flume and Kafka.

Explore how distributed file systems split data into blocks, replicate them across nodes, and manage metadata to enable fault-tolerant, scalable big-data processing with Hadoop and Spark.

Understand how the NameNode and daemons manage Hadoop storage, blocks and replicas, including metadata, data locality, and high availability through master-slave orchestration.

Understand how the Hadoop job tracker orchestrates block placement, replicas, and metadata through a fault-tolerant scheduling system, balancing load and monitoring node health.

Explain the load balancing in Hadoop architecture 4, covering pre-balancing task selection, fault tolerance, health monitoring, and during-work speculative execution using jawbreaker, localizer, and nonlocal localizer strategies.

Learn how the namenode stores metadata and maps file blocks to replicas using the log and fsimage, with the secondary namenode providing checkpoints to speed restarts.

Explore Hadoop Yarn architecture, including resource manager, application master, and node managers, and learn how containers are allocated through scheduling, with high availability and fault tolerance.

Demonstrate how a mapper transforms input text into word-count key-value pairs within a Hadoop MapReduce workflow. Learn to configure, implement, and submit the job.

Explore how the reducer aggregates values by key in MapReduce, producing final key-value output after grouping. Learn about input/output formats and driver setup for the reducer workflow.

Explore MapReduce aggregations, including average, max, min, and count, using map and reduce phases, with grouping by department number and sex.

Explore how MapReduce performs multiple aggregations and groupings, including count, average, max, and min, across single-column and multi-column groupings such as department number and sex.

Demonstrate mapper-only mapreduce workflows by applying filters, transformations, and column selections on unstructured data, while suspending reducers and producers when no aggregation is needed.

Explore using reduce in MapReduce to perform grouping and aggregation. Build bidirectional friends lists from social graph data.

Spark is an execution framework for big data that uses in-memory computing and a dag-based model to run transformations in parallel, with persistence and flexible partitions to boost speed.

Explore spark components, including core spark, spark streaming, graph processing with spark graphics, and MLlib for machine learning. Learn how these in-memory tools connect with data warehouses and real-time analytics.

Explore spark architecture: data is partitioned across a cluster, transformed via map and other operations, with actions triggering lazy evaluation and a distributed dataflow across executors.

Explore spark rdd caching and persistence, including in-memory and disk storage, flow control during floor execution, and how persistent rdds survive subsequent actions.

Learn how to create RDDs in Spark from local data and text files, apply map, flatMap, filter, and reduce, and perform a word count with partitioned aggregations.

Explore aggregations in Spark by grouping data on multiple columns such as department number and sex, and compute metrics like average, maximum, and minimum.

Spark prog 3 shows how to perform multiple aggregations with grouping and multi-level grouping, applying max, min, average, sum, and count across data.

Demonstrates merging datasets with unions, handling duplicates with distinct, and applying Cartesian products via cross joins to compare data across departments and sales in Spark.

this lecture introduces scala as a functional programming language with object-oriented features, and covers immutable vs mutable variables, collections, and map-filter-append patterns in the repl.

Learn Scala map collections by transforming elements, accessing keys and values, and filtering with boolean conditions using map and flatMap, including uppercase transformations and concise shortcuts.

Learn Scala 3 transformations over collections, including map, flatMap, slice, and filter, and see how Spark processes distributed data with last-expression semantics and grouped aggregations.

Explore Scala transformations in Spark using conditional and structural transformations, map and match cases to transform lists, records, and strings, with practical examples from salary, departments, and grades.

Learn how to transform records into key-value pairs or case classes for Spark aggregations, perform group by operations, and convert to schemaful data frames for Spark SQL.

Explore Scala fundamentals in a functional and imperative style, defining power and factorial recursively, practicing type casting, collections, case match, and Spark style reductions with cumulated operations.