Big Data Foundation for Developers

Master the Hadoop framework and HDFS storage component in the big data foundation for developers course by examining design principles, architecture, and a case study on storing data.

Explore how big data uses distributed processing across a fault-tolerant cluster with data redundancy and locality, and how Hadoop's HDFS storage and Map Reduce processing handle data and computation.

Explore the HDFS design principles for handling petabyte-scale data with fault tolerance, write-once read-many semantics, and parallel processing; learn block storage, replication, and namenode and datanode roles.

Explore the HDFS master work architecture with a single name node on the master and data nodes on the workers, where clients access data directly from data nodes.

Demonstrates how hdfs stores a 300 megabyte file as five blocks on five data nodes with a 64 megabyte block size and a 3x replication to provide fault tolerance.

Understand hdfs placement: rack awareness, three replicas—first on a random node, second on a different rack, and third on the same rack as the second, on a different machine.

Explore how HDFS distributes blocks across racks, covering 64 MB blocks, replication factor 3, and locality rules that store the first copy on the same node when a process runs.

Explore how HDFS provides interfaces to access the file system, including the fs shell (hdfs dfs) for creating, adding, retrieving, and appending files, dfsadmin, and a web interface for namenode.

Install Hadoop 3.2.0 on a Linux machine by verifying Java 1.7+ and ssh, installing pdsh, downloading and extracting the tarball, configuring HADOOP_HOME, PATH, and hadoop-env.sh, and refreshing the shell.

Configure hdfs by editing core-site.xml and hdfs-site.xml and set fs.default.fs to hdfs://localhost:9000. Change the replication factor to 2, format the namenode, and start the hdfs daemons with start-dfs.sh.

verify hadoop servers run with jps, create /user/trainer1 in HDFS, copy aTale.txt, run wordcount via hadoop jar, and inspect the web UI at port 9870 to confirm setup.

Explore essential HDFS commands to copy files from local to HDFS, list permissions and replication, and retrieve data with cat and get, using HDFS dfs and Hadoop fs options.

Copy files from hdfs to local using hdfs dfs -get or hadoop fs -get, with optional local path. Understand hdfs root and /user/username home directories and permissions.

Learn hdfs dfs commands like ls, chmod, cp, append, mkdir, mv, rm -r, and rmdir for managing files, and understand setrep to change replication factor and how namenode handles copies.

Explore how data nodes store blocks and per-block checksums, while the name node tracks the filesystem via fsimage and edits, and how heartbeat and block reports keep it updated.

The secondary name node is the HDFS checkpoint process that rolls the edit log and updates fsimage. It enables quick recovery by providing a current fsimage if namenode fails.

Explore HDFS, the distributed storage system of Hadoop, including block distribution, replication, and rack-aware placement, plus commands to store and retrieve data, and prepare for map reduce processing.

Explore the Hadoop mapreduce framework and its architecture, including stages and daemons, and write Java mapreduce code to compile a jar. Practice using the combiner stage in code.

Explore the processing layer of Hadoop by examining MapReduce, following mastery of HDFS. Learn how MapReduce handles data processing within the Hadoop ecosystem.

Apply map reduce to process distributed data with Hadoop, using HDFS for locality and automatic fault tolerance. Scale massively with thousands of parallel tasks on a shared-nothing design.

Explore a case study of distributed processing across five factories, where local managers count workers and sum results, illustrating distributed, parallel, and data local processing.

Explore distributed processing by counting error messages across five machines, with local counts on each machine and a single global sum reaching twelve million.

Explore how map reduce processes big data by splitting it into input splits, performing parallel map tasks, and using independent reduce tasks to produce the final outputs.

Understand the master-worker model in map reduce, with a central job tracker handling jobs and task trackers on workers that run mapper and reducer tasks in parallel across a cluster.

Explore how map reduce works from job submission with Hadoop APIs to parallel map tasks, shuffle and sort, and final reduce tasks that aggregate results, plus partitioner and combiner.

Explain how map tasks process HDFS blocks into key-value pairs, using independent mappers for each block, configurable input formats, and filtering techniques for word counts and errors.

Demonstrate a MapReduce workflow on a two-part system log: two mappers extract the third field (log type) and output (type,1), while the reducer sums counts for INFO, WARNING, and ERROR.

Demonstrates a simple java mapper in Hadoop MapReduce reading log lines as longWritable and text, filtering for info, warning, error, emitting log type as key and intwritable 1 via context.

Explore how the reducer processes mapper output into key and list of values, sums each key's values, and writes results to separate files in parallel.

Extend the base reducer class to handle text keys and int or long writable values, sum values per key, and write results to separate files for each reducer via context.write.

Explore map reduce across industries with wordcount, log type counting, stock min and max, and telecom minutes totals, using horizontal and vertical filtering and aggregation.

Shuffle and sort collects mapper outputs, sorts and groups by key, and routes each key and its list of values to exactly one reducer, ensuring keys arrive in sorted order.

Understand the map reduce master-worker architecture with job tracker on the master and task tracker on workers, using heartbeats, data locality, and HDFS integration for fault-tolerant scheduling.

Explore how Hadoop components connect, from the master node with name node and job tracker to worker nodes with data nodes, task trackers, mappers, and reducers.

Explore how input formats decide map reduce splits by reading key-value pairs with a record reader, from text and key-value text to nline, combine file, and sequence file formats.

Discover how a combiner reduces network traffic by aggregating mapper output in mapreduce on HDFS, sending a single summed value to the reducer and using a commutative, associative function.

Explore how map reduce stages produce mapper output, combiner output, shuffle and sort, and reducer output through a hands-on puzzle with two mappers and one reducer.

Study the map reduce driver class for the LogProcess Java job, wiring mapper and reducer, enabling a combiner, two reducers, and command line file input/output paths for Hadoop.

Learn to compile MapReduce program using yarn: set the classpath with yarn, compile LogProcess.java with javac, package into LogProcess.jar, and prepare a yarn jar submission for hdfs input and output.

Demonstrates reviewing a mapreduce program, compiling the LogProcess.java, creating a jar, and preparing for a Hadoop mapreduce run by setting classpath and including a combiner.

Explore the map-reduce paradigm for big data processing, detailing mappers, reducers, and the shuffle-and-sort stage, plus the combiner and share-nothing parallelism, and preview yarn scheduling for hadoop processing.

Explore yarn, Hadoop's processing component, and learn to manage cluster resources with capacity scheduler; set up pseudo distributed single-node cluster to run map reduce jobs and monitor via web UI.

Explore Hadoop's core components—HDFS for storage and MapReduce for processing—and learn how Yarn's architecture optimizes cluster resource management for MapReduce jobs.

Learn how Yarn governs CPUs and memory resources on large clusters, ensuring production users get required resources and efficient allocation for development and test workloads.

Explore Yarn architecture: a master resource manager on the cluster coordinates resource allocation with node managers on each worker node, running applications inside containers led by an application master.

Explore how the Yarn resource manager's scheduler allocates memory, cpu, disk, and network resources at the application or queue level, and how the applications manager starts and restarts application master.

Explore how Yarn’s scheduler allocates memory and CPU resources, enforces administrator constraints, and supports capacity and fair schedulers for multi-tenant clusters.

Capacity scheduler enables multi-tenant sharing of a single cluster by enforcing hierarchical queues and capacity guarantees for memory and cpu, meeting SLAs for departments with preemptive scheduling and access control.

Node managers report resource availability to the resource manager and launch containers on node. The application master assigns container tasks, while nodemanager checks health and sends reports every 2 minutes.

Discover how the application master negotiates resources with the resource manager, requests CPUs and memory, and runs tasks in containers. Monitor tasks, restart failed tasks, and release containers upon completion.

Configure and run Yarn on a Linux Hadoop cluster, edit mapred-site.xml and yarn-site.xml, start resource and node managers, submit a Yarn job, and monitor via the Yarn web interface.

Submit the mapreduce job to yarn using the LogProcess.jar, verify the hdfs input, and inspect the reducer outputs in outdir showing 3 errors, 6 info, and 1 warning.

Set up the Yarn scheduler, run Hadoop programs with Yarn, and verify outputs. Learn Yarn architecture, capacity scheduler, and how it supports Spark and Hive on Hadoop.

Discover hive and its workflow for big data, install and connect to hive, and process data with hive using SQL queries and Beeline. Learn about hive metastore, partitions, and buckets.

Explore Hive features as a SQL interface to MapReduce, using HQL and ANSI SQL-92 style queries on data in HDFS, with a metastore managing table metadata and MapReduce jobs.

Move your data warehouse to big data Hive by creating schema-matching tables, loading into HDFS, and running map reduce queries with filters, aggregations, and transformations, storing results in Hive.

Create a stocks table with ticker, value, and status; load data from stocks.csv on HDFS; and Hive runs MapReduce to filter status good and select max value per ticker.

Download and set up Hive on Linux by installing a stable Hive release, configuring Hive home and path, creating HDFS directories, and initializing the Derby metastore with the schema tool.

Initialize the Hive schema with Derby using schematool, then start HiveServer2 as a background daemon and verify with jps and netstat on port 10000.

Connect to hive using beeline, create a stocks table in hive, load data from hdfs/stocks.csv, and compute the max value per ticker for good status, with jobs running on yarn.

Learn how the hive metastore stores tables, columns, partitions, and metadata, and compare embedded, multi-user database, and separate service modes for scalable, secure access with spark and others.

Learn how to access and query hive data using the beeline command line interface, connect to hive server2, and manage queries with semicolons, quit commands, and history.

Explore Hive data model basics: tables as data units stored as flat files on HDFS, partitioned by values, optionally bucketed, with partitions excluded from table columns.

Discover how Hive partitions data by year into separate directories for targeted queries, and how bucketing by ticker_name into 100 buckets stored within each partition speeds access.

Access HDFS data as a Hive external table by location; Hive does not copy data, creates only the schema in the metastore.

Developers gain practical exposure to Hive basics, Beeline access, creating and querying tables, and partitioning and bucketing concepts, paving the way to processing data with Spark.

Explore Spark fundamentals, install and start Spark shell, and practice Scala basics to develop big data applications, including loading, transforming, and storing data, and applying machine learning.

Explore Spark, an Apache platform for processing data with beyond map and reduce paradigms, offering processing, sql, machine learning, graph libraries, and interfaces in Java, Scala, Python, R, and sql.

Download spark 2.4.4 prebuilt for Hadoop 2.7+ with wget and tar, then unzip and move to the spark directory, and configure spark-env.sh for Hadoop, Yarn, and Hive using hive-site.xml.

Kill the hiveserver2 process, verify with jps to ensure no Hive connections remain, then start the Spark shell to connect to Hive using Spark.

Practice spark-shell with yarn master, set log level to error, and use sc and spark session for rdd and sql queries, show databases and tables, including stocks and group-by result.

Explore Scala as a high level language on the Java virtual machine that blends object oriented and functional programming, with type inference and Java interoperability for scalable parallelism.

Explore a simple Scala hello world example using an object with a main method that prints Hello World in spark-shell, highlighting no semicolons and case sensitivity.

Explore Scala language components such as objects, classes, fields, and methods, with public private protected access, illustrated by an intStack example using push, pop, and the double colon operator.

Explore Scala's basic data types—byte, int, short, long, float, double, char, string, boolean, unit, null, nothing, and any—along with mutable var, immutable val, type inference, and tuples.

Explore Scala operators, mirroring Java with assignment operators like += and -=, arithmetic, relational, logical, and bitwise operations. Note that Scala lacks the increment and decrement operators ++ and --.

Explore Scala statements that support procedural code within a functional language: use assignment, if-else, for and while loops, returns, blocks in curly braces, and function definitions.

Explore while and do-while loops with pre and post checks and at-least-once execution, and for loops using to, until, by, and print versus println.

Master functions in Scala as the core of a functional language, using def syntax, treating functions as objects or methods, and understanding return types and last-expression results.

Define anonymous functions in Scala using parameters and => as function literals or variables; explore function2 and other arities. See how closures capture external variables and determine the result.

Explore Scala collections and arrays, declaring arrays with the Array keyword and types in square brackets, iterating with for loops, printing with printf, and using length and plus for concatenation.

Explore Scala collections such as lists, sets, maps, tuples, options and iterators. See immutable-by-default behavior, 0-based indexing, head, tail, prepend, append, and operations like plus, keys, values, contains and get.

Define classes with fields and methods and instantiate multiple objects using new. Extend classes, reuse toString, and explore singleton objects with the object keyword, shown by a stack.

Companion objects in Scala share class name and reside in the same file, used to initialize the class with an apply method called when the object is invoked with parentheses.

Explore scala case classes, which encapsulate fields, automatically provide companion objects and methods like apply, copy, equals, and toString, and enable powerful pattern matching for spark datasets.

Learn how traits template classes and support multiple implementations, enabling type parameterization with element type T. Build a string-to-type map and derive string and int stack classes from the trait.

Explore placeholders and underscores in Scala to implement higher order and anonymous functions. Learn to pass functions as parameters and use placeholder syntax for concise code in Spark processing.

Learn Scala basics and Spark integration with Hadoop Yarn and Hive for map-reduce based big data processing. Explore key Scala constructs and their use with Spark.

Explore how Spark distributes data as RDDs across a cluster, applying parallelized transformations, then shift to dataframe based processing with Hive and window operations.

Spark uses a master driver and worker executors in a master worker architecture, where the driver allocates tasks, monitors progress, caches data across executors, and works with Yarn.

Explore resilient distributed datasets (RDDs) and their in-memory, immutable, distributed nature, with spill-to-disk support and Spark lineage for rebuilding data when a node fails.

Spark uses RDDs for data processing, including transformations such as map, reduce, filter, and group by, and actions like count, collect, and save, with distributed execution.

Boosts performance by caching data in memory and disk blocks, preserving data locality, and using a lazy operator graph to enable fault-tolerant recomputation from the source RDD.

Explore loading a log file into an RDD, using Spark shell in local mode, and applying map, filter, and reduceByKey to count and summarize log types in HDFS.

Explore common Spark RDD operations, including map, reduce, filter, groupBy, sortBy, flatMap, and joins, with emphasis on parallel execution and data aggregation.

Explore how rdd actions trigger lazy evaluation in spark, executing transformation plans and producing results; cover reduce, collect, count, first, take, saveAsTextFile, and foreach, with move toward dataframes and datasets.

Explore the Spark sql interface to access tables and run parallel operations on data, powered by a distributed sql engine with Hive ql support for hive-like sql on Spark.

Explore dataframes as distributed collections of structured data with named columns; create them from hive tables, Json or parquet files, or RDDs with a schema, and manipulate data like rows.

Read stocks.csv from HDFS into a dataframe in spark-shell, and inspect its schema with printSchema and show. Use projections, apply-based column access, and price>100 filters to illustrate lazy evaluation.

Explore Apache Parquet, a column-oriented storage format, and Spark's read and write support with optional compression. Learn predicate pushdown to reduce disk reads.

Spark interfaces with Hive to access data in Hive tables. Link hive-site.xml in the Spark conf directory and use spark.sql to return results as a dataframe.

Explore creating and querying Spark data frames from HDFS CSV files with header and inferred schema, and using Parquet with filters, predicate push down, and column operations.

Read and join stock dataframes from parquet and csv files, perform a ticker-based join to attach names, then apply distinct, union, and except operations with a broadcast hash join.

Explore how transformations and actions operate on data frames in Spark, using dfp and dfn from stock data to perform count, collect, distinct, head, take, foreach, and reduce.

Learn how dataframe operations split into transformations and actions. Transformations keep data distributed and run on executors, while actions bring results to the driver, so scale remains parallel.

Explore dataframe transformations in spark by applying filter, select, where, and groupBy to manipulate data; learn orderBy, withColumn, join, union, except, sample, and distinct for scalable analytics.

Master dataframe actions and lazy evaluation by triggering computations with actions. Use collect, count, first, take, head, show, foreach, and reduce on Row objects, with commutative and associative reduction.

Learn aggregate operations on dataframes with Spark, using relational grouped dataset via groupBy and agg for multiple metrics, and pivoting to convert row results into columns, with Oracle offering options.

Explore grouping and pivoting dataframes by ticker and status, performing count, sum, and max aggregations, with aliases to name results and handling nulls when no rows exist.

Explore window operations in Spark to compute per-group summaries while preserving row-level context. Use window specifications with partition by, order by, and rowsBetween to perform deduplication and aggregate counts.

Use window functions on stock OHLC data to compute monthly open and monthly close by partitioning by ticker and month, with row numbers for the first day of each month.

Dataframe parallelism depends on partitions and spark.sql.shuffle.partitions, not total cores. Partitions derive from file size, block size, and storage; set property=value for spark.sql.files.maxPartitionBytes and spark.sql.shuffle.partitions, and check via spark.conf.getAll.

Create a temporary view from a data frame to query with spark.sql in the current session, acting as a non-materialized placeholder, and learn about the global_temp view accessible across sessions.

Cache dataframe or table with spark.catalog.cacheTable or dataframe.cache to speed up repeated access, caching occurs on first access due to lazy evaluation, with spills to disk if memory is insufficient.

Access the Spark web interface on port 4040 of the master node or localhost. View jobs, stages, task counts, and SQL plans to optimize performance and avoid sortmergejoin.

Explore datasets as a generalized form of dataframes in Spark, handling distributed data and supporting integers, strings, doubles, and case classes with field access and creation from objects or dataframes.

Convert a data frame to a dataset of stocks using a case class, filter and update values, union datasets, and create datasets from objects via toDS.

Learn how Spark sql provides a Hive-compatible query interface to run sql on dataframes, using spark.sql to create temp views and perform select, insert, and group by operations, returning dataframes.

practice sql queries on a spark dataframe by creating a temporary view tmpStocks, querying via sql, and optimizing shuffle partitions from 200 to 10 for faster execution.

Create and register user defined functions in spark sql, combine scala with distributed processing, and apply udfs to comma-separated data while considering serialization, performance, and sparing use.

Demonstrates creating and registering a Spark user defined function to remove duplicates from strings, using split, distinct, sort, and mkString, then applying it to a dataset via a temporary view.

Learn how spark runs on yarn by configuring yarn and hadoop conf dirs, submitting with spark-submit, and using cluster deploy mode with specified executors and cores.

Master spark dataframes and migrate data warehouse processing from rdbms to spark. Explore spark architecture with driver and executors, yarn, rdds, sql, transformations, actions, aggregations, window functions, hive, and udfs.

Explore how big data and Spark enable machine learning with algorithms like decision tree, linear regression, k-means, and collaborative filtering, and build a machine learning pipeline to evaluate model accuracy.

Explore machine learning as a branch of artificial intelligence, where algorithms learn from data to make reliable predictions and enable online recommendations.

Train models with sample data in supervised learning to predict results. Explore unsupervised learning and anomaly detection, and examine reinforcement learning for achieving goals in economic models.

Explore machine learning approaches, including supervised classification and regression, unsupervised clustering, collaborative filtering, and dimensionality reduction, with examples like decision trees, logistic and linear regression, Naïve Bayes, k-means, ALS, PCA.

Explore spark's dataframe based machine learning library, including classification, regression, clustering, collaborative filtering, and dimensionality reduction, with built in statistics, sampling, hypothesis tests, and data conversion utilities.

Learn how ml algorithms use features as vectors and produce labels, using spark ml.linalg.Vector for n-dimensional data, including sparse and dense vectors and setFeaturesCol and setLabelCol to rename columns.

Identify categorical (discrete) and numerical (continuous) features, such as color and temperature, and learn how Spark converts categories with a string indexer while preserving numeric ordering.

Explore a simple spark-based machine learning example using a decision tree to classify pets as good or not by weight, color, and legs, with about 80% accuracy and overfitting explained.

Explore how spark's estimators, transformers, and evaluators build, test, and tune a decision tree, including numeric conversions for strings and back, to optimize model accuracy.

Build a decision tree classifier in Spark on pet store data from hdfs, converting categorical features with indexers, assembling features, and evaluating accuracy on train/test splits.

Encode categorical features as one-hot vectors to avoid implying order. Spark's oneHotEncoderEstimator converts a category column into 0/1 vectors with fixed columns, illustrated by color examples.

Demonstrates using a one hot encoder to convert color and Good Pet into numeric vectors, then builds a decision tree classifier with VectorAssembler features and a training and test split.

Explore building a Spark pipeline that automates feature conversion, string-to-integer conversion, vector assembly, estimator-based training, and transforming data to predictions using stages, fit, and transform.

Build a spark ml pipeline with string indexer, assembler, and a decision tree, fitted with all data. Split 80/20 for training and test, transform, and generate predictions on new data.

Learn linear regression as modeling Y with a linear combination of features X1 to Xn, using Spark methods like SGD, Lasso, ridge, and Gaussian on weather data.

Build a linear regression model using a Gaussian generalized linear regression on the Austin weather data to predict rain from features like humidity and wind velocity.

Explore anomaly detection to identify unusual patterns in data, such as fraud across transactions, logins, and data transfers, using unsupervised learning like k-means clustering.

Apply k-means clustering to detect anomalies by forming k clusters, defining centers as means, and assigning points by distance, then report top 1% clusters for fraud reduction.

Demonstrates kmeans clustering to detect anomalies in login data using k=3, transforming login into a features vector, and identifying the smallest cluster as the anomaly.

Explore collaborative filtering to build a recommendation engine using point-of-sale data from thousands of customers, including user ids and item details, to derive implicit ratings from purchase frequency.

Build a spark-based recommendation engine using ALS with implicit ratings, transforming item names to indices, and generating four personalized recommendations from retail purchase data.

Build practical machine learning skills by implementing decision trees, linear regression, k-means, and collaborative filtering, and reinforce learning through plenty of exercises.

Become a proficient big data developer by writing map reduce programs, transforming data frames to parquet, applying windowing and pivoting, and leveraging machine learning and UDF.