Apache Spark and Scala

Discover Scala, a hybrid functional and object-oriented language that runs on the Java Virtual Machine, offering type inference, var and val mutability, and Java interoperability from main method onward.

Learn scala basics by using the scala interpreter to run expressions, write a hello world program with an object and main method, and compile and run from the command line.

Discover how Scala classes and objects define blueprints, create objects with new, and access fields and methods in a program that demonstrates a move method and a constructor with parameters.

Learn how to use if and else statements in Scala, evaluate boolean expressions, and handle nested conditions with practical examples that show when code executes under true or false.

Explore Scala loops, including while, do-while, and for loops, with examples showing top- and bottom-checked conditions and iterating over a list.

Explore functions and procedures in Scala, define with def, support nested definitions and special character names, and create procedures with no return value; learn calling and compiling Scala.

Explore how private, protected, and public access modifiers control visibility in Scala, including inner classes, package scopes, and default public access.

Explore how Scala uses Java strings as immutable objects, create and concatenate them, and work with arrays, including multidimensional arrays and value assignments.

Explore Scala collections, including lists, sets, and maps, and compare immutable and mutable variants while using operations like head, tail, isEmpty, and map methods.

Scala traits bridge single inheritance and multiple inheritance by enabling mixins; they define method signatures and may provide methods, offering interfaces like reuse for Java classes that use abstract classes.

Explore Scala pattern matching, the language's most widely used feature, through examples that show case statements, default handling, and console output when values match integers or objects.

Explore Scala extractors and the apply method, illustrated through practical examples inside the Spark and Scala course, reinforcing functional pattern ideas.

Explore scala's exception handling with try, catch and finally, throwing a legal argument exception and pattern matching in catch blocks for file not found and input/output exceptions.

Learn scala file IO by writing to and reading from files using the java.io package. Read console input with readLine and display file content with the source class.

Learn programming with RDDs in Apache Spark, where resilient distributed datasets are immutable and created from files or programmatically across a cluster.

Discover how spark shell enables distributed data analysis across many machines. Create and configure a spark context—the main entry point for initializing a spark application—and manage a single active context.

Create rdds by loading an external dataset or by parallelizing a collection in the driver, using SparkContext's textFile for external data and parallelize for in-memory collections.

Explore RDD operations in Spark, comparing transformations that build new RDDs (such as map and filter) with actions that produce final results or write to storage, and understand lazy evaluation.

Explore the lifecycle of a Spark program from loading data via parallelize or storage to applying lazy transformations such as map, caching data, and launching actions like count or collect.

Learn spark caching for in-memory data, comparing raw caching and serialized caching for small versus larger data sets, and using persist to avoid recomputation while managing memory and garbage collection.

Explore common Spark transformations and actions, including map and filter transformations, and practical examples of collect, count, take, reduce, and fold on an RDD to transform and aggregate data.

Explore Spark functions in Scala by loading data with the text file method and parallelize method, then apply map, flat map, and map partitions transformations, and finally collect the results.

Explore advanced Spark transformations, including map partitions with index, set name, max and min, and save as object file.

Create and manipulate key-value pairs in Spark by loading data via the text file method, applying map to split lines by comma, and extracting customer id as the key.

Explore Spark aggregate functions, including group by key, reduce by, and aggregate by, with practical word-count examples and sorting by key.

Explain and apply aggregate functions in Spark with key-value pairs, including count by key, group by, and aggregate by key, using map, reduce, and collect to produce sorted results.

Explore join operators in Spark, including inner, left outer, and right outer joins on pair RDDs, plus combineByKey summing and actions like countByKey and lookup.

Create a Spark word count in Scala by loading a text file, splitting with flatMap, mapping to (word, 1), reducing by key, and optionally sorting before collect.

Learn to use Spark shared variables, especially accumulators and broadcast variables, to aggregate values from nodes back to the driver, count events, and track blank lines in data pipelines.

Spark handles fault tolerance by launching a speculative copy of slow tasks when nodes lag or crash. Accumulators must be used in actions to ensure reliable updates despite multiple evaluations.

Use broadcast variables to send read-only data, such as lookup tables or feature vectors, to all Spark nodes efficiently. Treat them as immutable and distributed in memory to optimize closures.

Explore numeric RDD operations in Spark, including count, mean, max, min, variance, sum, and standard deviation, demonstrated by loading data and applying these functions.

Learn per-partition operations in Spark with map partitions, map partitions with index, and foreach partition to minimize per-element setup, while exploring partition IDs and per-partition value sums.

Explore Spark runtime architecture, detailing the driver–executor master-slave model, cluster resource allocation, and how transformations and actions execute as tasks.

Learn how the Spark driver runs your program, creates the Spark context, and translates user code into transformations and actions, then schedules tasks on executors.

Explain how executors run your transformations and tasks, launched at the start of a Spark application and lasting lifetime, with driver coordinating results and block manager providing in memory storage.

Apache Spark uses pluggable cluster managers (YARN, Mesos, standalone, EMR) to launch drivers and executors via spark-submit, with resource scheduling and deployment options.

Understand Spark cluster managers: standalone uses a master with workers and configurable memory and cores; YARN enables shared resources on Hadoop; Mesos offers dynamic resource sharing in its default mode.

Spark sql integrates structured data with a schema and enables data access across databases. Load from sources, work with data frames and data streams, and optimize queries with catalyst.

Initialize the Spark and SQL context as the starting point for working with structured data, enabling creation of data objects and execution of SQL queries, with table caching and management.

Discover spark sql caching and three in-memory options for caching tables: hive context, fixed objects, and direct cache calls; learn to cache, uncache, and verify cached status.