Learn Apache Spark and Scala from Scratch

Explore functions and procedures in Scala, including functions vs methods, definitions, units, and how to call them, illustrated by an add int example returning the sum.

Explore Scala's call by name parameters, where a code block is passed and evaluated each time the parameter is accessed, delaying computation versus by value parameters.

Explore how Scala handles functions with variable arguments, enabling variable length argument lists. The demo prints hello, Scala, and Python and shows compiling and running the program.

Explore arrays in Scala, including declaring fixed-size arrays, indexing, multi-dimensional arrays, concatenation, and processing elements with loops and simple examples.

Explore scala collections, including lists, sets, maps, tuples, and options. Understand mutable vs immutable, and remember to start with immutable collections before switching to mutable.

Discover sets in Scala, including immutable and mutable variants, and learn basic operations like head, tail, and isEmpty, plus concatenation with ++ and min, max, and intersections.

Learn how options in Scala handle some or none values using the option type, get, get or else, and pattern matching, with isEmpty checks and Java null comparisons.

Explore Scala pattern matching, including case alternatives and arrows, match test with integers and case classes, and compare pattern matching to Java switch in practical demos.

Explore Scala traits as a midway point between Java interfaces and multiple inheritance, enabling mixing traits into classes, partial implementations, and guidelines for when to use traits versus abstract classes.

Discover the big data overview, definitions, and the four v's: velocity, volume, variety, veracity, plus the data types structured, semi-structured, and unstructured.

Explore the facts about big data, showing how social networks, mobile devices, and video platforms generate massive volumes—from tweets per minute to petabytes per year.

Explore how big data drives faster time to market and improved customer experiences, highlighting risk analysis, fraud detection, churn prevention, and recommendation engines across financial, retail, telecom, and government sectors.

Explore how top Hadoop users like Amazon Web Services, IBM Infosphere BigInsights, Cloudera, MapR, and Datastax Enterprise harness Hadoop based software to manage big data cost-effectively and enable enterprise analytics.

Explore how RDBMS and Hadoop differ in scaling (monolithic vs parallel commodity servers), storage (tables with schema vs key-value pairs), and use cases (online transactions vs offline big data streaming).

Explore the three Hadoop cluster modes—standalone (local), pseudo distributed, and fully distributed—and their uses in MapReduce, development, QA, and production.

Explore hdfs daemons—namenode, data nodes, and secondary name node—for reliable, high-throughput storage with block replicas, and mapreduce daemons—job tracker and task tracker—for distributed processing with heartbeat monitoring.

Explore Hadoop cluster architecture, including client, master (name node, secondary name node, job tracker), and slave (data node, task tracker) roles for storing and processing unstructured data with MapReduce.

Explore why learning Hadoop accelerates big data analytics, opens abundant career opportunities, and complements Spark and other Hadoop ecosystem tools for scalable data processing.

Explore Hadoop distributions and their compatibilities, including Cloudera distribution for Hadoop, CDH, HDP, and MapR, and learn how they resolve version incompatibilities across operating systems such as Ubuntu.

Explore the Hadoop ecosystem and its core components, including HDFS, MapReduce, Yarn, Hive, Pig, HBase, Hcatalog, and more, with insights into their roles and data processing capabilities.

Explore Hadoop distributions and compatibilities, learn how they resolve version conflicts, and review options like Cloudera, CDH, HDP, MapR, with Ubuntu, CentOS, RedHat.

Explore how HDFS splits files into 64 MB blocks, stores them on data nodes under the NameNode, and uses threefold replication with rack awareness to ensure fault tolerance.

Explore HDFS components and architecture, including the name node as master and data nodes for block storage; learn about blocks, replication, FS image, edit logs, and heartbeat coordination.

Explore Apache Spark, a fast, general-purpose cluster engine that handles batch, iterative, interactive queries, and streaming in memory, with simple Python, Java, Scala, and SQL APIs.

Learn the job execution flow and Spark execution, contrasting Hadoop's disk-based data access with Spark's in-memory processing and noting cache steps and deployment options Mesos, Yarn, or Spark's cluster manager.

Explore spark architecture, including resilient distributed datasets and directed acyclic graphs, and see how driver, executors, and cluster managers orchestrate transformations and actions.

Learn how to create Apache Spark RDDs using parallelized collections, external datasets, and transforming existing RDDs, with practical examples loading text, CSV, and JSON data.

Explore RDD transformations in Spark, creating new RDDs from input RDDs with immutable data, lazy execution, and a DAG lineage, including map, filter, and flatMap across narrow and wide transformations.

Explore spark caching fundamentals by comparing raw memory caching and serialized caching, highlighting faster processing versus lower memory usage for iterative workloads.

Learn how to perform joins in Spark, including left and right outer joins, with paired RDDs, combined by key, and actions like count by key, collect, and lookup.

Learn how broadcast variables in Spark share large read-only data, like lookup tables or feature vectors, across worker nodes efficiently using sc.broadcast and accessed via .value.

Discover how per partition operations in Spark reduce repeated setup by running code once per partition, using map partitions, map partitions with index, and for each partition on an RDD.

Learn how Spark SQL integrates with Spark programs, enabling SQL and DataFrame queries, streaming computation, uniform data access, connectivity, and user defined functions across Hive, Avro, Parkwood, ORC, JSON, JDBC.

Learn to perform advanced spark sql queries with the dataset api, including creating a case class, converting rdds to datasets, reading json, and filtering employees aged 18 to 30.

Explore Spark Streaming as an extension of the core Spark API for scalable, real-time, fault-tolerant live data processing, and examine a Twitter sentiment analysis use case.

Explore the fundamentals of spark streaming, including streaming context setup and D streams. Apply transformations such as map, flatMap, filter, reduce, and groupBy, with caching, accumulators, broadcast variables, and checkpoints.

Explore Apache Spark graphics, the graph API for graphs and graph parallel computation, including its ETL integration, exploratory analysis, and a flight data analytics use case.

Explore graphs as mathematical structures and abstract data types, objects connected by relations, represented by vertices and edges, including undirected and directed graphs with edge attributes like cost or label.

Explore Spark GraphX, the Spark API for graphs and graph-parallel computation, featuring property graphs built on RDDs, a directed multigraph with parallel edges and user-defined vertex and edge properties.

Discover GraphX features that unify ETL, enable flexible graph and collection processing, and support RDD-based transforms and iterative algorithms via the Freegal API, with PageRank and triangle count.