Master Apache Spark (Scala) for Data Engineers

Discover how Apache Spark, a distributed in-memory framework, delivers 100x faster parallel processing and includes Spark SQL with dataframes, Spark Streaming, MLlib, and GraphX.

Understand how spark applications run by studying cluster managers like yarn, the master-slave cluster design, and how resource and node managers allocate resources and launch containers on hdfs.

Explore distributed storage and the Hadoop distributed file system (HDFS). See how files divide into 128 MB blocks stored across a cluster, managed by a name node and data nodes.

Submit your spark application on yarn/hdfs, let the application master negotiate resources for executors, and process data blocks stored in hdfs under the name node's validation.

Explore deployment modes, client mode where the driver runs on the client machine, and cluster mode where the driver runs inside the application master, with cluster mode recommended for production.

Set up Spark on a local machine by installing Java and JAVA_HOME, downloading the Spark binary, extracting it, and configuring PATH and Hadoop binaries, then verify with Spark commands.

Understand how the driver starts the app, checks syntax, builds a logical plan and a physical plan, then assigns tasks to executors who register with the driver.

Compare RDDs and dataframes in Spark, create an RDD from a file and a dataframe from JSON with multiline, and see dataframes' table-like representation and improved performance.

Launch the spark shell in local mode with -master local[1] and verify the spark UI at localhost:4040 shows the driver and executors.

Run the spark-shell on a cluster with yarn as the master, create three executors with six cores, and show how commands load data frames across the cluster and return results.

Explore how Spark transformations create new data frames from existing ones (and data sets) while remaining lazy until an action triggers execution, with filters and shows illustrating immutability.

Learn how narrow transformations avoid data shuffling while wide transformations, such as group by, require shuffling and create stages, illustrating how data is shuffled across executors to group by year.

Learn how actions trigger Spark jobs and how stages and tasks form. See how wide transformations cause shuffles and spark.shuffle.partitions, shaping the DAG-based execution plan.

Call explain on a data frame to reveal logical to optimized logical to physical plans, showing how json file scanning, filtering, and group by drive stage transitions in Spark.

Explore how to create a Hadoop cluster on Google Cloud Dataproc using the Dataproc API, and compare it with your local Spark setup.

Set up your environment on google cloud by creating a project, enabling data proc APIs. Launch a one-master, zero-worker big data cluster with spark, hive, and hadoop.

Learn to run Spark jobs on Google Cloud Platform using Spark shell or zipline notebooks, initializing Spark and SparkSession variables, and launching notebooks for interactive data processing.

Explore how to use the hdfs file system in a google cloud data cluster, including uploading files, creating folders, moving data, and retrieving with get.

Explore reading json files with the Spark data frame API, including multiline json, merging schemas from multiple jsons, reading from a directory, and writing results with overwrite mode.

Learn to load Parquet data in Spark, inspect its schema, create a data frame, filter records by doctor id, and write results back to Parquet while understanding columnar, Snappy compression.

Read CSV into a Spark dataframe with and without headers, apply a custom schema, and write the results with compression options.

Explore reading and writing avro data frames in spark, infer schemas, and use avro tools with schemas stored with the data, highlighting binary formatting and compression.

Learn to read xml data into a spark dataframe using the spark xml library, inspect schema and sample records, and save the data back to xml with updated tags.

Explore how to modify Spark dataframes by changing column data types, updating values with substring, adding derived or literal columns, and dropping or renaming columns for clean data.

Master string manipulation in Spark using dataframe APIs: uppercase first names, lowercase last names, concatenate with ascorbic_w, replace via regexp, and split emails to extract domains.

Learn to convert a string date column to date or timestamp, then extract year, month, day, and add current date or timestamp in a Spark dataframe.

Learn to use the data frame filter API in Spark for conditional predicates, such as email contains a value, country in a list, and id between two values.

Learn spark sql by registering a data frame as a temporary view named customers, then run sql queries like select with order by and use group by and sql operators.

Read a Hive table from the Hive metastore into a Spark data frame, run Spark SQL queries on top, and write the results back to Hive as a new table.

Learn to set up IntelliJ IDEA community edition, install the Scala plugin, restart the IDE, and begin writing Spark applications in Scala.

Set up a Scala Spark project from IntelliJ, choose sbt, align Scala and Spark versions, and add Spark dependencies (core, sql, hive) to build a runnable Spark application.

Write your first spark program on the IDE by creating a serializable spark driver and a spark session entry point. Use SparkSession.builder to get or create a session with configurations.

Customize a spark session with a custom configuration by setting the application name and master, run locally with three threads, and use key-value, config object, or runtime file methods.

Read a multiline json file into a Spark dataframe, filter out adventure movies, uppercase titles, group by release year with collect_list, and write results back to json.

Explore how a Spark application's execution plan yields jobs, stages, and tasks, and learn to use the Spark UI to inspect their details and shuffle partition effects.

Compare spark dataframe and dataset, and show how to convert between them for flexible workflow. Demonstrate how using case classes and implicits provides compile-time safety in dataset operations like filtering.

Create and apply a user defined function in Spark to categorize ratings as top_rated or low_rated, read a JSON dataset, register the function, and add a rating category column.

Learn to run an Apache Spark application on Amazon cloud using an EMR cluster and explore how AWS enables big data analytics with Spark and Hadoop.

Learn to set up a first-year EMR spark cluster on AWS, including choosing m5.xlarge instances, configuring a 3-node master/worker setup, key pairs, and secure remote access via PuTTY.

Learn to run spark on EMR with spark-submit, read input from S3, write output to EMR storage, and set IAM roles for S3 access.

Monitor spark applications with the spark history UI and the timeline server on EMR. Learn to control executors, cores, and memory during spark-submit, including disabling dynamic allocation.

Submit a spark application to an EMR cluster with spox submit, using a jar from S3 and the main class, in client mode, and verify output in DFS.

Learn how the map transformation in spark applies a function to every element of a text file, creating a doubled result, with execution only when a collect action runs.

The lecture outlines environment options for Spark: Cloudera VM for a local Hadoop cluster and the local Spark setup, noting the 16 GB RAM requirement and that Cloudera is optional.

Compare the map and flatMap transformations, showing how flatMap turns each input element into multiple outputs by splitting lines into words, resulting in a final nine-element dataset.

Master the filter and intersection transformations in Spark, using predicates to select even numbers and ages under 40, and find common elements between datasets.

Apply the distinct transformation to remove duplicates from a collection, and use union to combine two in-memory lists, then collect to view the combined result.

GroupByKey in Spark groups data by a key, triggering a shuffle and a new stage. Map lines to key-value pairs, then apply groupByKey to aggregate by month.

The lecture explains reduceByKey in Spark, using an accumulator and value to combine counts across partitions and illustrate a shuffle, with example computing total students per subject (science, math, computer).

Sort data in Spark using sortByKey and sort by transformations, demonstrating ascending and descending orders. Convert numeric fields to integers for correct order and sort by specific values like scores.

Master how Spark partitions a distributed data set to enable parallel processing, and use mapPartitions and mapPartitionsWithIndex to run per-partition logic, such as creating a database connection per partition.

Learn to adjust Spark partitions with coalesce and repartition, decreasing partitions with coalesce to avoid reshuffling, and increasing partitions with repartition, which may reshuffle data.

Explore the join transformation in Apache Spark, aligning records by a key such as the name, and learn how inner, left outer, and right outer joins work.

Learn how common spark actions drive execution, including collect, take, first, count, top, count by value, reduce, and foreach, with practical demonstrations and memory considerations.