Apache Spark Core and Structured Streaming 3.0 In-Depth

Explore how Apache Spark serves as a fast, memory-first unified computing engine for batch, streaming, machine learning, and graph processing, with a consistent API across libraries.

Explore how resilient distributed datasets (RDDs) partition data, apply transformations, and use lineage for fault-tolerant, lazy evaluation, with actions triggering parallel Spark computations across a cluster.

Explore Spark architecture, including Spark context, driver, executors, and cluster managers, and learn how RDDs, partitions, caching, and jars enable distributed processing.

Explore how to write our first Spark program using map, filter, and reduce on RDDs, and understand transformations, actions, parallelism, and lazy evaluation.

Download and install Apache Spark using local or cloud options, set up PyCharm or Databricks, launch a single node or cluster, and explore the Spark session and Spark context.

Discover Spark data aggregation using a weather CSV to form year-temperature pairs, reduce by key to find yearly maximums, and understand partitions, shuffle, and lazy evaluation.

Explore how a Spark program reads temperature data, applies map and reduce by key, and uses narrow and wide transformations, partitions, and hash partitioning to trigger jobs, stages, and tasks.

Explore how partitioners assign each data item to a partition during Spark transformations, triggering shuffles. Compare hash partitioner and range partitioner, including skew and range-based sorting.

Explore non-aggregating operations on RDDs from creation and partitioning to filtering, mapping, and sorting, then use actions like take and collect and optimize with map partitions.

Delve into aggregations on RDDs with map values, flatMapValues, reduce by key, group by key, combine by key, cogroup, and subtract by key, plus related actions like collect and lookup.

Learn to perform word count and maximum temperature aggregations on pair RDDs in Spark using flatMap, split, and reduceByKey or countByValue.

Join two rdds to explore inner, left outer, and right outer joins, illustrated with a weather and zip city example that computes city max temperatures in spark core and streaming.

Discover broadcast variables in Spark: distribute a read-only dictionary from driver to executors, cache it on each node, and perform fast joins with no shuffle using in-memory lookups.

Explore how to use broadcast variables to perform a map-only inner join between NYSE daily stock data and dividends, enabling dividend-aware stock price analysis on Spark.

Cache and persist rdds to keep in-memory data across actions, using storage levels to balance memory and disk, improving iterative processing and preventing repeated shuffles.

Explore caching in Apache Spark Core while analyzing a healthcare dataset to quantify stroke risk by work type, gender, and age group using RDDs and reduce by key.

Learn how accumulators, distributed variables in Spark, tally word counts and log metrics across partitions with driver-side aggregation, while noting retry recomputation caveats.

Explore Spark's combineByKey through a credit card transaction analysis, calculating total expenditure per user, per user-card, and per user-card-category, plus distinct categories and maximum expenditure by user.

Explore secondary sort in Apache Spark Core with the Python API by analyzing car loan data, ranking monthly loan requests, top car prices, and location-based top models.

Learn to run Spark on a multi-node cluster using aws emr, configure security, submit jobs with yarn in cluster mode, and analyze New York taxi data on hdfs.

Learn how to size spark executors for optimal throughput by balancing memory, cores, and executor count. It covers configuration methods, memory models, and yarn overhead.

Explore dynamic allocation in Spark, letting the cluster adjust executors based on pending tasks and idle time, while using an external shuffle service to preserve shuffle data.

Learn to handle skewed data in spark core and streaming by repartitioning partitions. Use combiners, salting, and two-phase aggregation, plus adaptive execution in spark sql for skewed keys.

Explore Spark tuning tactics to reduce empty partitions, apply coalesce to lower partitions without shuffles, cache big data with selective repartitioning, and optimize joins with small datasets.

Explore a lead data engineer's journey into big data and Spark. Learn core skills, Spark Core and Spark SQL concepts, and practical optimization tips for building a data engineering career.