Big Data Foundation for Developers

Explore the HDFS master work architecture with a single name node on the master and data nodes on the workers, where clients access data directly from data nodes.

Demonstrates how hdfs stores a 300 megabyte file as five blocks on five data nodes with a 64 megabyte block size and a 3x replication to provide fault tolerance.

verify hadoop servers run with jps, create /user/trainer1 in HDFS, copy aTale.txt, run wordcount via hadoop jar, and inspect the web UI at port 9870 to confirm setup.

Explore essential HDFS commands to copy files from local to HDFS, list permissions and replication, and retrieve data with cat and get, using HDFS dfs and Hadoop fs options.

Apply map reduce to process distributed data with Hadoop, using HDFS for locality and automatic fault tolerance. Scale massively with thousands of parallel tasks on a shared-nothing design.

Understand the master-worker model in map reduce, with a central job tracker handling jobs and task trackers on workers that run mapper and reducer tasks in parallel across a cluster.

Explore how Hadoop components connect, from the master node with name node and job tracker to worker nodes with data nodes, task trackers, mappers, and reducers.

Study the map reduce driver class for the LogProcess Java job, wiring mapper and reducer, enabling a combiner, two reducers, and command line file input/output paths for Hadoop.

Learn to compile MapReduce program using yarn: set the classpath with yarn, compile LogProcess.java with javac, package into LogProcess.jar, and prepare a yarn jar submission for hdfs input and output.

Node managers report resource availability to the resource manager and launch containers on node. The application master assigns container tasks, while nodemanager checks health and sends reports every 2 minutes.

Configure and run Yarn on a Linux Hadoop cluster, edit mapred-site.xml and yarn-site.xml, start resource and node managers, submit a Yarn job, and monitor via the Yarn web interface.

Submit the mapreduce job to yarn using the LogProcess.jar, verify the hdfs input, and inspect the reducer outputs in outdir showing 3 errors, 6 info, and 1 warning.

Discover hive and its workflow for big data, install and connect to hive, and process data with hive using SQL queries and Beeline. Learn about hive metastore, partitions, and buckets.

Create a stocks table with ticker, value, and status; load data from stocks.csv on HDFS; and Hive runs MapReduce to filter status good and select max value per ticker.

Learn how the hive metastore stores tables, columns, partitions, and metadata, and compare embedded, multi-user database, and separate service modes for scalable, secure access with spark and others.

Learn how to access and query hive data using the beeline command line interface, connect to hive server2, and manage queries with semicolons, quit commands, and history.

Discover how Hive partitions data by year into separate directories for targeted queries, and how bucketing by ticker_name into 100 buckets stored within each partition speeds access.

Access HDFS data as a Hive external table by location; Hive does not copy data, creates only the schema in the metastore.

Download spark 2.4.4 prebuilt for Hadoop 2.7+ with wget and tar, then unzip and move to the spark directory, and configure spark-env.sh for Hadoop, Yarn, and Hive using hive-site.xml.

Kill the hiveserver2 process, verify with jps to ensure no Hive connections remain, then start the Spark shell to connect to Hive using Spark.

Explore Scala language components such as objects, classes, fields, and methods, with public private protected access, illustrated by an intStack example using push, pop, and the double colon operator.

Explore Scala statements that support procedural code within a functional language: use assignment, if-else, for and while loops, returns, blocks in curly braces, and function definitions.

Define anonymous functions in Scala using parameters and => as function literals or variables; explore function2 and other arities. See how closures capture external variables and determine the result.

Explore Scala collections and arrays, declaring arrays with the Array keyword and types in square brackets, iterating with for loops, printing with printf, and using length and plus for concatenation.

Learn how traits template classes and support multiple implementations, enabling type parameterization with element type T. Build a string-to-type map and derive string and int stack classes from the trait.

Explore placeholders and underscores in Scala to implement higher order and anonymous functions. Learn to pass functions as parameters and use placeholder syntax for concise code in Spark processing.

Spark uses a master driver and worker executors in a master worker architecture, where the driver allocates tasks, monitors progress, caches data across executors, and works with Yarn.

Explore resilient distributed datasets (RDDs) and their in-memory, immutable, distributed nature, with spill-to-disk support and Spark lineage for rebuilding data when a node fails.

Boosts performance by caching data in memory and disk blocks, preserving data locality, and using a lazy operator graph to enable fault-tolerant recomputation from the source RDD.

Explore common Spark RDD operations, including map, reduce, filter, groupBy, sortBy, flatMap, and joins, with emphasis on parallel execution and data aggregation.

Explore the Spark sql interface to access tables and run parallel operations on data, powered by a distributed sql engine with Hive ql support for hive-like sql on Spark.

Spark interfaces with Hive to access data in Hive tables. Link hive-site.xml in the Spark conf directory and use spark.sql to return results as a dataframe.

Explore how transformations and actions operate on data frames in Spark, using dfp and dfn from stock data to perform count, collect, distinct, head, take, foreach, and reduce.

Learn how dataframe operations split into transformations and actions. Transformations keep data distributed and run on executors, while actions bring results to the driver, so scale remains parallel.

Explore dataframe transformations in spark by applying filter, select, where, and groupBy to manipulate data; learn orderBy, withColumn, join, union, except, sample, and distinct for scalable analytics.

practice sql queries on a spark dataframe by creating a temporary view tmpStocks, querying via sql, and optimizing shuffle partitions from 200 to 10 for faster execution.

Demonstrates creating and registering a Spark user defined function to remove duplicates from strings, using split, distinct, sort, and mkString, then applying it to a dataset via a temporary view.

Learn how spark runs on yarn by configuring yarn and hadoop conf dirs, submitting with spark-submit, and using cluster deploy mode with specified executors and cores.

Explore how big data and Spark enable machine learning with algorithms like decision tree, linear regression, k-means, and collaborative filtering, and build a machine learning pipeline to evaluate model accuracy.

Explore machine learning approaches, including supervised classification and regression, unsupervised clustering, collaborative filtering, and dimensionality reduction, with examples like decision trees, logistic and linear regression, Naïve Bayes, k-means, ALS, PCA.

Learn how ml algorithms use features as vectors and produce labels, using spark ml.linalg.Vector for n-dimensional data, including sparse and dense vectors and setFeaturesCol and setLabelCol to rename columns.

Explore a simple spark-based machine learning example using a decision tree to classify pets as good or not by weight, color, and legs, with about 80% accuracy and overfitting explained.

Encode categorical features as one-hot vectors to avoid implying order. Spark's oneHotEncoderEstimator converts a category column into 0/1 vectors with fixed columns, illustrated by color examples.

Build a spark ml pipeline with string indexer, assembler, and a decision tree, fitted with all data. Split 80/20 for training and test, transform, and generate predictions on new data.

Build a linear regression model using a Gaussian generalized linear regression on the Austin weather data to predict rain from features like humidity and wind velocity.