Big Data Foundation for Developers

Master the Hadoop framework and HDFS storage component in the big data foundation for developers course by examining design principles, architecture, and a case study on storing data.

Explore how big data uses distributed processing across a fault-tolerant cluster with data redundancy and locality, and how Hadoop's HDFS storage and Map Reduce processing handle data and computation.

Explore the HDFS design principles for handling petabyte-scale data with fault tolerance, write-once read-many semantics, and parallel processing; learn block storage, replication, and namenode and datanode roles.

Explore the HDFS master work architecture with a single name node on the master and data nodes on the workers, where clients access data directly from data nodes.

Demonstrates how hdfs stores a 300 megabyte file as five blocks on five data nodes with a 64 megabyte block size and a 3x replication to provide fault tolerance.

Explore how HDFS provides interfaces to access the file system, including the fs shell (hdfs dfs) for creating, adding, retrieving, and appending files, dfsadmin, and a web interface for namenode.

Install Hadoop 3.2.0 on a Linux machine by verifying Java 1.7+ and ssh, installing pdsh, downloading and extracting the tarball, configuring HADOOP_HOME, PATH, and hadoop-env.sh, and refreshing the shell.

Configure hdfs by editing core-site.xml and hdfs-site.xml and set fs.default.fs to hdfs://localhost:9000. Change the replication factor to 2, format the namenode, and start the hdfs daemons with start-dfs.sh.

verify hadoop servers run with jps, create /user/trainer1 in HDFS, copy aTale.txt, run wordcount via hadoop jar, and inspect the web UI at port 9870 to confirm setup.

Explore essential HDFS commands to copy files from local to HDFS, list permissions and replication, and retrieve data with cat and get, using HDFS dfs and Hadoop fs options.

Copy files from hdfs to local using hdfs dfs -get or hadoop fs -get, with optional local path. Understand hdfs root and /user/username home directories and permissions.

Learn hdfs dfs commands like ls, chmod, cp, append, mkdir, mv, rm -r, and rmdir for managing files, and understand setrep to change replication factor and how namenode handles copies.

The secondary name node is the HDFS checkpoint process that rolls the edit log and updates fsimage. It enables quick recovery by providing a current fsimage if namenode fails.

Explore HDFS, the distributed storage system of Hadoop, including block distribution, replication, and rack-aware placement, plus commands to store and retrieve data, and prepare for map reduce processing.

Explore the processing layer of Hadoop by examining MapReduce, following mastery of HDFS. Learn how MapReduce handles data processing within the Hadoop ecosystem.

Apply map reduce to process distributed data with Hadoop, using HDFS for locality and automatic fault tolerance. Scale massively with thousands of parallel tasks on a shared-nothing design.

Understand the master-worker model in map reduce, with a central job tracker handling jobs and task trackers on workers that run mapper and reducer tasks in parallel across a cluster.

Explore how map reduce works from job submission with Hadoop APIs to parallel map tasks, shuffle and sort, and final reduce tasks that aggregate results, plus partitioner and combiner.

Demonstrate a MapReduce workflow on a two-part system log: two mappers extract the third field (log type) and output (type,1), while the reducer sums counts for INFO, WARNING, and ERROR.

Explore how the reducer processes mapper output into key and list of values, sums each key's values, and writes results to separate files in parallel.

Extend the base reducer class to handle text keys and int or long writable values, sum values per key, and write results to separate files for each reducer via context.write.

Explore map reduce across industries with wordcount, log type counting, stock min and max, and telecom minutes totals, using horizontal and vertical filtering and aggregation.

Shuffle and sort collects mapper outputs, sorts and groups by key, and routes each key and its list of values to exactly one reducer, ensuring keys arrive in sorted order.

Understand the map reduce master-worker architecture with job tracker on the master and task tracker on workers, using heartbeats, data locality, and HDFS integration for fault-tolerant scheduling.

Explore how Hadoop components connect, from the master node with name node and job tracker to worker nodes with data nodes, task trackers, mappers, and reducers.

Explore how input formats decide map reduce splits by reading key-value pairs with a record reader, from text and key-value text to nline, combine file, and sequence file formats.

Study the map reduce driver class for the LogProcess Java job, wiring mapper and reducer, enabling a combiner, two reducers, and command line file input/output paths for Hadoop.

Learn to compile MapReduce program using yarn: set the classpath with yarn, compile LogProcess.java with javac, package into LogProcess.jar, and prepare a yarn jar submission for hdfs input and output.

Demonstrates reviewing a mapreduce program, compiling the LogProcess.java, creating a jar, and preparing for a Hadoop mapreduce run by setting classpath and including a combiner.

Explore Hadoop's core components—HDFS for storage and MapReduce for processing—and learn how Yarn's architecture optimizes cluster resource management for MapReduce jobs.

Explore Yarn architecture: a master resource manager on the cluster coordinates resource allocation with node managers on each worker node, running applications inside containers led by an application master.

Explore how Yarn’s scheduler allocates memory and CPU resources, enforces administrator constraints, and supports capacity and fair schedulers for multi-tenant clusters.

Capacity scheduler enables multi-tenant sharing of a single cluster by enforcing hierarchical queues and capacity guarantees for memory and cpu, meeting SLAs for departments with preemptive scheduling and access control.

Node managers report resource availability to the resource manager and launch containers on node. The application master assigns container tasks, while nodemanager checks health and sends reports every 2 minutes.

Configure and run Yarn on a Linux Hadoop cluster, edit mapred-site.xml and yarn-site.xml, start resource and node managers, submit a Yarn job, and monitor via the Yarn web interface.

Submit the mapreduce job to yarn using the LogProcess.jar, verify the hdfs input, and inspect the reducer outputs in outdir showing 3 errors, 6 info, and 1 warning.

Discover hive and its workflow for big data, install and connect to hive, and process data with hive using SQL queries and Beeline. Learn about hive metastore, partitions, and buckets.

Create a stocks table with ticker, value, and status; load data from stocks.csv on HDFS; and Hive runs MapReduce to filter status good and select max value per ticker.

Download and set up Hive on Linux by installing a stable Hive release, configuring Hive home and path, creating HDFS directories, and initializing the Derby metastore with the schema tool.

Connect to hive using beeline, create a stocks table in hive, load data from hdfs/stocks.csv, and compute the max value per ticker for good status, with jobs running on yarn.

Learn how the hive metastore stores tables, columns, partitions, and metadata, and compare embedded, multi-user database, and separate service modes for scalable, secure access with spark and others.

Learn how to access and query hive data using the beeline command line interface, connect to hive server2, and manage queries with semicolons, quit commands, and history.

Explore Hive data model basics: tables as data units stored as flat files on HDFS, partitioned by values, optionally bucketed, with partitions excluded from table columns.

Discover how Hive partitions data by year into separate directories for targeted queries, and how bucketing by ticker_name into 100 buckets stored within each partition speeds access.

Access HDFS data as a Hive external table by location; Hive does not copy data, creates only the schema in the metastore.

Explore Spark fundamentals, install and start Spark shell, and practice Scala basics to develop big data applications, including loading, transforming, and storing data, and applying machine learning.

Explore Spark, an Apache platform for processing data with beyond map and reduce paradigms, offering processing, sql, machine learning, graph libraries, and interfaces in Java, Scala, Python, R, and sql.

Download spark 2.4.4 prebuilt for Hadoop 2.7+ with wget and tar, then unzip and move to the spark directory, and configure spark-env.sh for Hadoop, Yarn, and Hive using hive-site.xml.

Kill the hiveserver2 process, verify with jps to ensure no Hive connections remain, then start the Spark shell to connect to Hive using Spark.

Practice spark-shell with yarn master, set log level to error, and use sc and spark session for rdd and sql queries, show databases and tables, including stocks and group-by result.

Explore a simple Scala hello world example using an object with a main method that prints Hello World in spark-shell, highlighting no semicolons and case sensitivity.

Explore Scala language components such as objects, classes, fields, and methods, with public private protected access, illustrated by an intStack example using push, pop, and the double colon operator.

Explore Scala statements that support procedural code within a functional language: use assignment, if-else, for and while loops, returns, blocks in curly braces, and function definitions.

Define anonymous functions in Scala using parameters and => as function literals or variables; explore function2 and other arities. See how closures capture external variables and determine the result.

Explore Scala collections and arrays, declaring arrays with the Array keyword and types in square brackets, iterating with for loops, printing with printf, and using length and plus for concatenation.

Define classes with fields and methods and instantiate multiple objects using new. Extend classes, reuse toString, and explore singleton objects with the object keyword, shown by a stack.

Learn how traits template classes and support multiple implementations, enabling type parameterization with element type T. Build a string-to-type map and derive string and int stack classes from the trait.

Explore placeholders and underscores in Scala to implement higher order and anonymous functions. Learn to pass functions as parameters and use placeholder syntax for concise code in Spark processing.

Spark uses a master driver and worker executors in a master worker architecture, where the driver allocates tasks, monitors progress, caches data across executors, and works with Yarn.

Explore resilient distributed datasets (RDDs) and their in-memory, immutable, distributed nature, with spill-to-disk support and Spark lineage for rebuilding data when a node fails.

Spark uses RDDs for data processing, including transformations such as map, reduce, filter, and group by, and actions like count, collect, and save, with distributed execution.

Boosts performance by caching data in memory and disk blocks, preserving data locality, and using a lazy operator graph to enable fault-tolerant recomputation from the source RDD.

Explore common Spark RDD operations, including map, reduce, filter, groupBy, sortBy, flatMap, and joins, with emphasis on parallel execution and data aggregation.

Explore the Spark sql interface to access tables and run parallel operations on data, powered by a distributed sql engine with Hive ql support for hive-like sql on Spark.

Spark interfaces with Hive to access data in Hive tables. Link hive-site.xml in the Spark conf directory and use spark.sql to return results as a dataframe.

Read and join stock dataframes from parquet and csv files, perform a ticker-based join to attach names, then apply distinct, union, and except operations with a broadcast hash join.

Explore how transformations and actions operate on data frames in Spark, using dfp and dfn from stock data to perform count, collect, distinct, head, take, foreach, and reduce.

Learn how dataframe operations split into transformations and actions. Transformations keep data distributed and run on executors, while actions bring results to the driver, so scale remains parallel.

Explore dataframe transformations in spark by applying filter, select, where, and groupBy to manipulate data; learn orderBy, withColumn, join, union, except, sample, and distinct for scalable analytics.

Explore grouping and pivoting dataframes by ticker and status, performing count, sum, and max aggregations, with aliases to name results and handling nulls when no rows exist.

Explore window operations in Spark to compute per-group summaries while preserving row-level context. Use window specifications with partition by, order by, and rowsBetween to perform deduplication and aggregate counts.

Use window functions on stock OHLC data to compute monthly open and monthly close by partitioning by ticker and month, with row numbers for the first day of each month.

Dataframe parallelism depends on partitions and spark.sql.shuffle.partitions, not total cores. Partitions derive from file size, block size, and storage; set property=value for spark.sql.files.maxPartitionBytes and spark.sql.shuffle.partitions, and check via spark.conf.getAll.

Create a temporary view from a data frame to query with spark.sql in the current session, acting as a non-materialized placeholder, and learn about the global_temp view accessible across sessions.

Access the Spark web interface on port 4040 of the master node or localhost. View jobs, stages, task counts, and SQL plans to optimize performance and avoid sortmergejoin.

Learn how Spark sql provides a Hive-compatible query interface to run sql on dataframes, using spark.sql to create temp views and perform select, insert, and group by operations, returning dataframes.

practice sql queries on a spark dataframe by creating a temporary view tmpStocks, querying via sql, and optimizing shuffle partitions from 200 to 10 for faster execution.

Create and register user defined functions in spark sql, combine scala with distributed processing, and apply udfs to comma-separated data while considering serialization, performance, and sparing use.

Demonstrates creating and registering a Spark user defined function to remove duplicates from strings, using split, distinct, sort, and mkString, then applying it to a dataset via a temporary view.

Learn how spark runs on yarn by configuring yarn and hadoop conf dirs, submitting with spark-submit, and using cluster deploy mode with specified executors and cores.

Explore how big data and Spark enable machine learning with algorithms like decision tree, linear regression, k-means, and collaborative filtering, and build a machine learning pipeline to evaluate model accuracy.

Train models with sample data in supervised learning to predict results. Explore unsupervised learning and anomaly detection, and examine reinforcement learning for achieving goals in economic models.

Explore machine learning approaches, including supervised classification and regression, unsupervised clustering, collaborative filtering, and dimensionality reduction, with examples like decision trees, logistic and linear regression, Naïve Bayes, k-means, ALS, PCA.

Learn how ml algorithms use features as vectors and produce labels, using spark ml.linalg.Vector for n-dimensional data, including sparse and dense vectors and setFeaturesCol and setLabelCol to rename columns.

Identify categorical (discrete) and numerical (continuous) features, such as color and temperature, and learn how Spark converts categories with a string indexer while preserving numeric ordering.

Explore a simple spark-based machine learning example using a decision tree to classify pets as good or not by weight, color, and legs, with about 80% accuracy and overfitting explained.

Build a decision tree classifier in Spark on pet store data from hdfs, converting categorical features with indexers, assembling features, and evaluating accuracy on train/test splits.

Encode categorical features as one-hot vectors to avoid implying order. Spark's oneHotEncoderEstimator converts a category column into 0/1 vectors with fixed columns, illustrated by color examples.

Build a spark ml pipeline with string indexer, assembler, and a decision tree, fitted with all data. Split 80/20 for training and test, transform, and generate predictions on new data.

Learn linear regression as modeling Y with a linear combination of features X1 to Xn, using Spark methods like SGD, Lasso, ridge, and Gaussian on weather data.

Build a linear regression model using a Gaussian generalized linear regression on the Austin weather data to predict rain from features like humidity and wind velocity.

Demonstrates kmeans clustering to detect anomalies in login data using k=3, transforming login into a features vector, and identifying the smallest cluster as the anomaly.

Build a spark-based recommendation engine using ALS with implicit ratings, transforming item names to indices, and generating four personalized recommendations from retail purchase data.

Become a proficient big data developer by writing map reduce programs, transforming data frames to parquet, applying windowing and pivoting, and leveraging machine learning and UDF.