Spark SQL and Spark 3 using Scala Hands-On with Labs

Set up a cloud nine environment via the cloud nine console, selecting ubuntu server 18.4, a t3.medium instance, and postgres with jupyter to practice data engineering.

Learn how to open ports on a Cloud9 instance to expose a web app, using DNS alias, IP, port 80, security group inbound rules, and elastic IP.

Learn to allocate and associate an elastic IP address with a Cloud9 instance to provide a stable DNS alias, handle changing IP addresses, and maintain access after reboots.

Increase Cloud9 storage from 10 gb to 32 gb by modifying the EBS volume in the EC2 console, reboot the instance, and verify with df -h.

Sign up for Google Cloud Platform to access a $300 credit for one year, set up a free trial, and provision virtual machines for hands-on learning.

Set up passwordless login on a single-node Hadoop cluster by generating SSH keys, copying the public key to authorized_keys, and validating passwordless access to start Hadoop components.

Start and validate HDFS on a single-node cluster by configuring the path and passwordless login, launching all DFS components, and validating with DFS commands and file operations.

Set up a single-node Hadoop cluster, start DFS and YARN, and validate the services. Clone the retail_db repo from GitHub, copy the data to HDFS, and verify readiness.

Download and install Spark 2.4.7 on a single-node Hadoop 2.7 cluster, extract the tarball, create a spark2 soft link, and run Spark via Python or Scala SQL for learning.

Learn to integrate Spark 2 with Jupyter Lab by creating a new kernel embedded with SPARC, configuring Yarn, and validating Spark and Hive databases on a single-node cluster.

Download Spark 3.1.1 for a single-node Hadoop and Spark cluster, unzip, move the folder, and create a symlink. Validate the installation by configuring and running basic scripts.

Validate spark 3 using clis by launching a single-node cluster, running Scala, Python, and SQL interfaces, and verifying access to retail_db.orders via Hive metastore and Spark queries.

Master Scala functions in Spark contexts by defining with def, using return types, and creating higher-order and anonymous functions to sum ranges, squares, cubes, and multiples.

Discover case classes, a boilerplate-reducing, immutable default construct that auto generates toString, equals, hashCode, copy, and product related utilities via a companion object.

Explore Scala collections: sequence, set, and map, with hands-on focus on list, set, and map APIs, iteration via foreach, and common traits like Traversable and Iterable for Spark integration.

Learn basic MapReduce in Spark with Scala, using map, filter, and reduce to sum squares of even numbers, and compare filter-first versus square-first strategies.

Set up datasets for basic I/O by cloning or downloading from GitHub, locate the data in the lab or local PC, then read files and perform MapReduce operations.

Read data from files using a source API, convert to in-memory collections, and apply map, filter, and reduce to compute order subtotals and total revenue.

Install sbt on Windows, download the appropriate version for Scala and Spark projects, and finish setup via command prompt. Launch sbt to verify first-time downloads complete.

Learn to list HDFS files with the dfs -ls command and options like -h for human-readable sizes, -R for recursive listing, and sort by name, time, or size.

Learn to create hdfs directories, set user space under /user, assign ownership to the login user, and adjust group ownership with dfs mkdir, -chown, and -chgrp, including recursive options.

Copy files from local to hdfs using copy from local (put); learn to create folders, preserve metadata, handle existing files with -f, and understand name node and block distribution.

Explore how to obtain file metadata in the Hadoop distributed file system using DFS commands, revealing files, blocks, and locations and explaining replication factor, block IDs, and data node mappings.

Learn how to read and override HDFS properties at runtime by inspecting core-site.xml and hdfs-site.xml. Use -D or --conf to set replication factor and block size for copied files.

Spark read APIs explain reading data into dataframes using csv, json, and text formats, with format, load, options, and schema to control headers, delimiters, and inferred schema.

Preview schema and data in Spark dataframes using print schema, show, and describe, with practical hands-on labs on Spark SQL and Spark 3 using Scala.

Discover Spark data frame APIs to read data into dataframes, apply standard and low-level transformations, filter, aggregate by group, sort, and project fields with select, drop, and withColumn.

Explore how to write Spark dataframes to multiple formats such as csv, json, and parquet using write APIs, options, compression, mode, and validation on a multi-node cluster.

Launch a Spark session in a notebook, create a Spark object with the Spark session builder, import implicits for shorthand column references to build dummy frame and explore SQL functions.

Master date and time arithmetic in spark using date_add, date_sub, add_months, and months_between. Learn to use current_date and current_timestamp with dataframe examples and understand end-of-month behavior.

Explore how to use Spark date_trunc and trunc to generate week-to-date, month-to-date, and year-to-date reports, and derive beginnings of date or time from timestamps with practical examples.

Explore basic transformations, including filtering, aggregation, and sorting, using data frame APIs on airlines data to compute daily totals of departure and arrival delays for one month.

Launch a Spark context by configuring a Spark session or Spark shell with local or yarn master settings, then explore filtering, aggregations, and sorting.

Explore filtering with spark data frame APIs on the airlines data, using sql style and dataframe style conditions with operators such as in, between, and like.

Filter Spark data frames to count flights with departure delay over 60 minutes, using sql-style and api-style methods, and preview results in the airlines dataframe (count = 40,104).

Count flights departing late on Sundays from the 2008 January data using Spark data frames in scala. Use date_format and to_date to extract day of week and filter late departures.