Master Apache Spark (Scala) for Data Engineers

Learn how actions trigger Spark jobs and how stages and tasks form. See how wide transformations cause shuffles and spark.shuffle.partitions, shaping the DAG-based execution plan.

Set up your environment on google cloud by creating a project, enabling data proc APIs. Launch a one-master, zero-worker big data cluster with spark, hive, and hadoop.

Read CSV into a Spark dataframe with and without headers, apply a custom schema, and write the results with compression options.

Set up a Scala Spark project from IntelliJ, choose sbt, align Scala and Spark versions, and add Spark dependencies (core, sql, hive) to build a runnable Spark application.

Read a multiline json file into a Spark dataframe, filter out adventure movies, uppercase titles, group by release year with collect_list, and write results back to json.

Learn to run spark on EMR with spark-submit, read input from S3, write output to EMR storage, and set IAM roles for S3 access.

GroupByKey in Spark groups data by a key, triggering a shuffle and a new stage. Map lines to key-value pairs, then apply groupByKey to aggregate by month.

Sort data in Spark using sortByKey and sort by transformations, demonstrating ascending and descending orders. Convert numeric fields to integers for correct order and sort by specific values like scores.

Master how Spark partitions a distributed data set to enable parallel processing, and use mapPartitions and mapPartitionsWithIndex to run per-partition logic, such as creating a database connection per partition.

Learn to adjust Spark partitions with coalesce and repartition, decreasing partitions with coalesce to avoid reshuffling, and increasing partitions with repartition, which may reshuffle data.

Explore the join transformation in Apache Spark, aligning records by a key such as the name, and learn how inner, left outer, and right outer joins work.