Apache Spark 2.0 with Java -Learn Spark from a Big Data Guru

Learn about resilient distributed datasets (RDDs) as Spark's core abstraction, and how to create, transform, and act on RDDs using map, filter, and count across a distributed cluster.

Create RDDs in Spark by parallelizing a collection with SparkContext or loading data from storage, with sources like local files, S3, HDFS, JDBC, Cassandra, and Elasticsearch.

Learn spark rdd transformations: filter selects data, map transforms elements, and returns a new rdd. See an airport data example filtering for united states and mapping to name and city.

Load airport data, parse latitude, filter latitudes above 40, map to airport names and latitudes, and save the results to an output file using Spark.

Explore flatMap transformation in Spark, producing multiple outputs per input via an iterator, and compare it with map in a wordcount example, including lambda usage.

Explore Spark set operations on RDDs, including sample and distinct, plus union, intersection, subtract, and Cartesian, with emphasis on shuffling, duplicates, and practical log-file examples.

Explore actions on RDDs in Apache Spark and how they return final results or persist data, including collect, count, take, save as text file, and reduce.

Implement sum of numbers solution in Spark: load prime file, apply flatMap to split by whitespace with a regular expression, filter empties, map to integers, and reduce to the sum.

Rdds are distributed across a cluster and partitioned for parallel processing. They are immutable and resilient, deterministic functions of their input that enable automatic recovery by Spark when nodes fail.

Demonstrates how RDDs use transformations and actions, with lazy evaluation delaying work until an action triggers execution, and how return types distinguish the two.

Spark persistence caches an RDD in memory or disk across actions, using storage levels (memory only, memory and disk, serialized or deserialized) to balance speed and memory, with LRU eviction.

Spark's core, sql, streaming, mllib, and graphx enable distributed batch and real-time processing with sql-like queries and scalable analytics.

Explore how pair RDDs store key-value data in Spark and support operations like counting by key and grouping by key. Create pair RDDs from key-value objects or by mapping IDs.

Create pair RDDs in Java by parallelizing a list of Scala tuples or mapping a regular RDD to key-value pairs using a pair function, resulting in persisted key-value data.

Apply filter and map values transformations to pair RDDs, preserve keys while changing values, and demonstrate with airports data including filtering by country and uppercasing names.

Explore reduce by key aggregation on paired RDDs to count word frequencies, using a (word, 1) map and a reducing function to sum values across keys.

Compute the average house price per number of bedrooms using Spark's reduceByKey on a paired RDD. Use an average count structure with total price and count to derive averages.

Group by key transforms a paired RDD by key into an iterable of values, enabling tasks like listing airports by country, and highlights when reduceByKey is preferable to avoid shuffling.

Explore the sort by key transformation in Spark with Java, using ascending and custom comparator options to sort pair RDDs, illustrated by sorting bedrooms and a word count task.

Demonstrates a Spark-based solution for the sorted word count problem by flipping key-value pairs, sorting by count in descending order, and flipping back to word-count pairs.

Leverage hash partitioning with a partitioner to keep the same keys on the same node, reduce shuffle, and persist results to avoid re-partitioning during joins and reduce by key.

Explore inner and outer join operations on pair RDDs, including left, right, and full outer joins, with emphasis on partitioning, shuffles, and best practices to avoid explosion of data.

Explore Spark accumulators to aggregate information across executors, using a Spark context to count total records, missing salary midpoint, and Canada records.

Add an accumulator to track bytes processed in a Spark job, register it with the Spark context, and update it from the Stack Overflow survey to print 2360248 bytes.

Broadcast variables cache a read-only lookup map on every worker, enabling efficient region lookups from postcodes without shipping large data with tasks.

Discover Spark SQL as the interface for structured and semi-structured data, and understand how a data frame and a dataset enable schema-aware, efficient queries in Spark 2.0.

Learn how Spark SQL operates inside a Spark application to load CSV data with a Spark session and data frame reader, then query with SQL-like operations and apply Catalyst Optimizer.

Apply Spark SQL to analyze house prices: load the CSV, cast price fields to integers, group by location, compute max price, then sort by average price per square feet.

Learn spark sql joins, including inner, left outer, right outer, and left semi joins, and how the catalyst optimizer improves join performance when mapping makerspace data by postcode to regions.

Learn to work with strongly typed datasets in Spark 2.0 using encoders to map Java objects, apply typed filters, rename and cast fields, and perform aggregations.

Compare dataset and rdd apis to decide when to use each, and learn how datasets enable sql, type safety, and catalyst optimization while rdds remain the core building block.

Learn to convert between dataset and rdd in spark using java, create a dataset from an rdd with a spark session, and print the schema and top 20 records.

Explore practical Spark SQL performance tuning by enabling pushdown, caching datasets with cache, and tuning in-memory columnar storage and batch size; optimize code generation to Java bytecode for longer queries.

Explore how Spark runs in cluster mode with a driver and executors, managed by a cluster manager. Learn to submit applications with spark-submit across standalone, YARN, and Mesos clusters.

Export your spark application to a jar with all dependencies using Gradle, then submit it to a local spark cluster via spark-submit in standalone mode.

Run a Spark application on an Amazon EMR cluster, load input data from S3, and submit the jar with spark submit to execute on the EMR Hadoop cluster.

Apply a data analytics workflow with Apache Spark to drive insights in big data projects. Engage with the Learning Big Data and Level Up group for ongoing tips and updates.