Spark SQL and Spark 3 using Scala Hands-On with Labs

Explore how to set up a cloud-based development environment with Cloud9, including creating an account, logging into the management console, and launching a cloud-based Python data engineering workspace.

Set up a cloud nine environment via the cloud nine console, selecting ubuntu server 18.4, a t3.medium instance, and postgres with jupyter to practice data engineering.

Explore the Cloud9 IDE, mastering terminals, file creation, and running code in a hands-on environment. Learn to configure preferences, fonts, explorer, GitHub integration, and built-in Docker for labs.

Explore how Cloud9 runs on an EC2 instance, including a 30-minute idle shutdown that saves costs. Access EC2 via the management console to review the Cloud9 lab demo instance.

Learn how to open ports on a Cloud9 instance to expose a web app, using DNS alias, IP, port 80, security group inbound rules, and elastic IP.

Learn to allocate and associate an elastic IP address with a Cloud9 instance to provide a stable DNS alias, handle changing IP addresses, and maintain access after reboots.

Increase Cloud9 storage from 10 gb to 32 gb by modifying the EBS volume in the EC2 console, reboot the instance, and verify with df -h.

Set up jupyter lab in cloud9 by validating python 3, creating and activating a virtual environment, installing jupyter lab, and launching on port 8888 with 0.0.0.0 binding for token-based access.

Sign up for Google Cloud Platform to access a $300 credit for one year, set up a free trial, and provision virtual machines for hands-on learning.

Explore the Google Cloud Platform web console to provision and manage virtual machines with Compute Engine, review billing details, and configure firewall rules for services deployed on VM instances.

Learn to estimate Google Cloud Platform costs with the pricing calculator for an Ubuntu VM with 4 CPU, 16 GB RAM, 60 GB storage, static IP, and pay-as-you-go concepts.

Provision and connect an ubuntu 18.04 lts virtual machine on Google Cloud Platform, configure a static external IP, and learn basic management through the web console.

Set up Docker on Ubuntu 18.04 in a cloud VM, install Docker, configure user permissions, and validate via hello-world, containers, and images.

Validate python on ubuntu servers, install pip, set up virtual environments, and install libraries like pandas and configparser, preparing for Jupyter deployment.

Set up a Ubuntu lab for Spark SQL and Spark 3 hands-on, configure a Python virtual environment, install Jupyter Lab, and configure GitLab and firewall rules for remote access.

Learn how to set up a single node Hadoop cluster on an Ubuntu 18.04 server, including prerequisites, Hadoop installation, DFS and YARN configuration and validation.

Set up prerequisites on a Google Cloud Ubuntu VM to prepare for a single-node Hadoop and Spark cluster, including wget and JDK installation, with browser-based and terminal access.

Set up passwordless login on a single-node Hadoop cluster by generating SSH keys, copying the public key to authorized_keys, and validating passwordless access to start Hadoop components.

Download, install, and configure Hadoop 3.3.0 on a Google Cloud provisioned single-node machine, start DFS components, and manage environment setup with Docker, Jupyter Lab, and gcloud.

Configure HDFS for a single-node Hadoop cluster by updating DFS XML and dot profile, determine Java home, format the DFS, and start the service.

Start and validate HDFS on a single-node cluster by configuring the path and passwordless login, launching all DFS components, and validating with DFS commands and file operations.

Configure a single node cluster by updating two xml files, including mapred-site.xml, for Hadoop. Copy, save, and start the components, then validate that they initialize properly.

start and validate yarn on a single-node Hadoop cluster by launching resource manager and node manager, verifying name node, data node, and hdfs readiness with passwordless login in place.

Manage a single-node Hadoop cluster on Ubuntu 18.4 by safely starting dfs and yan services, and validating name node, data node, resource manager, and node manager to control costs.

Set up a single-node Hadoop cluster, start DFS and YARN, and validate the services. Clone the retail_db repo from GitHub, copy the data to HDFS, and verify readiness.

Download and install Hive on top of Hadoop, configure and start a single-node cluster, version 3.1.2. Tar and extract the package, move it to location, and set up metastable db.

Set up a Postgres database for Hive Metastore using Docker on an Ubuntu 18.4 server, then validate connectivity and configure environment variables for seamless access.

Configure the Hive metastore to connect to a PostgreSQL database running in Docker, then initialize the schema to create Hive metastore metadata tables.

Launch and validate Hive by creating a retail DB, loading data into a table, and running queries to verify the setup on a single-node big data cluster.

Learn to manage a single node cluster with Hadoop, Hive, and a Postgres-backed Docker container by creating start_all and stop_all scripts to control DFS, YARN, and services, with validation commands.

Download and install Spark 2.4.7 on a single-node Hadoop 2.7 cluster, extract the tarball, create a spark2 soft link, and run Spark via Python or Scala SQL for learning.

Configure spark 2 by setting up hadoop paths, updating spark defaults, enabling the history server, creating spark logs and jobs folders, and integrating with hive metastore using the postgres driver.

Install and validate spark 2.4.7 with scala, python, and hive integration, using spark shell and spark sql to query hive metastore databases like retail db and orders.

Validate a single-node big data ecosystem by setting up a virtual environment, installing Jupyter Lab, and starting it for Hadoop and Spark use.

Learn to integrate Spark 2 with Jupyter Lab by creating a new kernel embedded with SPARC, configuring Yarn, and validating Spark and Hive databases on a single-node cluster.

Download Spark 3.1.1 for a single-node Hadoop and Spark cluster, unzip, move the folder, and create a symlink. Validate the installation by configuring and running basic scripts.

Configure Spark 3 on a single-node cluster by enabling Yarn, setting event log and hdfs directories, integrating with Hive metastore via Postgres, and preparing jars and xml configurations.

Validate spark 3 using clis by launching a single-node cluster, running Scala, Python, and SQL interfaces, and verifying access to retail_db.orders via Hive metastore and Spark queries.

Explore Spark 3 integration with Jupyter Lab using Python in a DataLab environment, create Spark kernel, and validate by running a Spark session and queries on a single node cluster.

Set up Scala for spark-based applications and learn Scala fundamentals for data processing. Install Scala, explore basic programming constructs and collections, and practice with the read-evaluate-print loop in the labs.

Install the Java JDK on Windows, download Scala binaries (2.12.4 or 2.10.6 as needed), install Scala, and verify by running Scala and hello world.

Explore basic programming constructs in Scala, including val and var, type inference, and string operations like upper, lower, and length. Build loops and conditionals using ranges and simple accumulators.

Master Scala functions in Spark contexts by defining with def, using return types, and creating higher-order and anonymous functions to sum ranges, squares, cubes, and multiples.

Explore object oriented concepts in Scala by building an Order class with constructor arguments and class variables, differentiating immutable val and mutable var, and generating getters, setters, and toString.

Explore Scala objects as singleton classes and companion objects with the same name as a class. Learn to define a main method and apply methods, invoked like static methods.

Discover case classes, a boilerplate-reducing, immutable default construct that auto generates toString, equals, hashCode, copy, and product related utilities via a companion object.

Explore Scala collections: sequence, set, and map, with hands-on focus on list, set, and map APIs, iteration via foreach, and common traits like Traversable and Iterable for Spark integration.

Learn basic MapReduce in Spark with Scala, using map, filter, and reduce to sum squares of even numbers, and compare filter-first versus square-first strategies.

Set up datasets for basic I/O by cloning or downloading from GitHub, locate the data in the lab or local PC, then read files and perform MapReduce operations.

Read data from files using a source API, convert to in-memory collections, and apply map, filter, and reduce to compute order subtotals and total revenue.

The lecture introduces tuples as a class-free data structure in Scala, enabling collections of values without named attributes. Access uses positional notation with underscores _1 to _6.

Develop a Scala application by following the development lifecycle: build and run a file, structure the project, create an object with a main function, and compile with sbt.

Learn to set up sbt, create build.sbt, compile scala code, package into a jar, and run via sbt run or scala.

Install sbt on Windows, download the appropriate version for Scala and Spark projects, and finish setup via command prompt. Launch sbt to verify first-time downloads complete.

Compile the Scala application into a jar with sbt and run it with arguments. Pass the first argument as order id and cast it to integer to drive the program.

Learn to set up IntelliJ IDEA for Scala development, install the Scala plugin and SBT, configure JDK, create a new Scala project, and enable auto import for dependencies.

Navigate the development cycle of a Scala app in IntelliJ, configure sbt, run with program arguments, compare sbt and Scala command flows, and package a jar for deployment.

Discover how to list and get help for HDFS commands from gateway nodes, connect to the name node (port 8020), and use core-site.xml to view its UI.

Learn to list HDFS files with the dfs -ls command and options like -h for human-readable sizes, -R for recursive listing, and sort by name, time, or size.

Learn to create hdfs directories, set user space under /user, assign ownership to the login user, and adjust group ownership with dfs mkdir, -chown, and -chgrp, including recursive options.

Copy files from local to hdfs using copy from local (put); learn to create folders, preserve metadata, handle existing files with -f, and understand name node and block distribution.

Copy files from hdfs to the local file system using get or copy to local, with patterns, multiple sources, and preserve metadata with -P while ignoring CRC.

Explore how to obtain file metadata in the Hadoop distributed file system using DFS commands, revealing files, blocks, and locations and explaining replication factor, block IDs, and data node mappings.

Learn to preview text data in hdfs files using commands like hadoop fs - tail and hadoop fs - cat, view first lines and headers, and list files recursively.

Learn how HDFS uses block size to divide files into blocks across nodes. Analyze how replication factor determines copies of each block across the cluster.

Explore how HDFS replication factor ensures data reliability by maintaining multiple copies of each block, with default factor three and rack awareness improving resilience.

Overview of checking hdfs storage usage with the df command, using -h for human readable sizes, and -s for summaries to reveal capacity, used, and free across retail db folders.

Explore hdfs stat commands to retrieve file metadata, including timestamp, size, type, block size, and replication factor, for directories and regular files.

Explore HDFS file permissions by interpreting rwx bits and octal mod values, and practice granting and revoking access for owner, group, and others, including 755 examples.

Learn how to read and override HDFS properties at runtime by inspecting core-site.xml and hdfs-site.xml. Use -D or --conf to set replication factor and block size for copied files.

Explore data processing with Spark 3 and Scala, read data from files, preview schemas, apply dataframe APIs, check duplicates, and write parquet data with compression within the data engineering lifecycle.

Launch spark-shell to start a spark session that creates the spark context automatically, and configure local or yarn cluster modes with master and port settings.

Spark read APIs explain reading data into dataframes using csv, json, and text formats, with format, load, options, and schema to control headers, delimiters, and inferred schema.

Preview schema and data in Spark dataframes using print schema, show, and describe, with practical hands-on labs on Spark SQL and Spark 3 using Scala.

Discover Spark data frame APIs to read data into dataframes, apply standard and low-level transformations, filter, aggregate by group, sort, and project fields with select, drop, and withColumn.

Master core Spark DataFrame functions to manipulate columns, including string, numeric, date, and aggregate operations, and derive new fields with select expressions, call, and lit.

Explore how to write Spark dataframes to multiple formats such as csv, json, and parquet using write APIs, options, compression, mode, and validation on a multi-node cluster.

Explore predefined Apache Spark SQL functions to derive new values from dataframe columns and apply select, filter, group by, and sort operations in lab environments.

Launch a Spark session in a notebook, create a Spark object with the Spark session builder, import implicits for shorthand column references to build dummy frame and explore SQL functions.

Create and explore a dummy dataframe in Spark using Scala, converting a list to a dataframe with toDF, applying current_date and alias for derived columns, and building an employees dataset.

Explore Spark dataframe column functions, focusing on string manipulation and date functions, including current date and current timestamp, to perform aggregates and data tasks with Spark SQL and Scala.

Demonstrate Spark special functions such as call and lead, and show how to pass columns as strings or as column types using dollar or call in data frame operations.

Learn how to use Spark special functions, especially lit, to convert strings to column literals for safe concatenation of first name and last name with a comma, avoiding type mismatches.

Learn to manipulate string columns in Spark SQL and Spark 3 using Scala by applying upper, lower, init cap, and length to nationality, using dollar or dataframe references and aliases.

Demonstrates using Spark's substring function to extract fixed-length data, such as last four digits of SSN and phone numbers, and first four digits of credit cards, with dataframe operations.

Learn to manipulate string columns with Spark's split function to extract delimited data, such as city, state, area code, and last four digits from phone numbers and SSNs.

Learn to concatenate strings in Spark using the concat function, handling variable arguments and inserting literals with lit, to create a full name from first and last names.

Pad strings with Spark functions to build fixed-length records using lpad and rpad. Pad numeric fields with zeros and non-numeric fields with hyphens or spaces to support month-end date concatenation.

Learn how to clean strings in Spark by trimming leading and trailing spaces and other characters using trim, ltrim, and rtrim, with practical examples on mainframe-derived fixture length records.

Explore Spark date and time functions by using current_date and current_timestamp, learn their server formats, and apply aliases and formatting in dataframe queries.

Master date and time arithmetic in spark using date_add, date_sub, add_months, and months_between. Learn to use current_date and current_timestamp with dataframe examples and understand end-of-month behavior.

Explore how to use Spark date_trunc and trunc to generate week-to-date, month-to-date, and year-to-date reports, and derive beginnings of date or time from timestamps with practical examples.

Explore Spark date and time functions, including date_format and year, month, day of week, day of month, and day of year, using dataframes and withColumn in Spark SQL.

Explore dealing with unix timestamp in spark, convert between unix timestamps and regular dates, and apply unix_timestamp and from_unixtime functions with format patterns for date and time.

Master over 300 predefined Spark SQL functions for string, date, and numeric transformations on dataframe columns, and apply them with select, filter, group by, and order by.

Explore basic transformations, including filtering, aggregation, and sorting, using data frame APIs on airlines data to compute daily totals of departure and arrival delays for one month.

Launch a Spark context by configuring a Spark session or Spark shell with local or yarn master settings, then explore filtering, aggregations, and sorting.

Explore filtering with spark data frame APIs on the airlines data, using sql style and dataframe style conditions with operators such as in, between, and like.

Explore filtering spark data frames by reading 2008 January data, inspecting schema, and selecting key columns like year, month, times, delays, and airports. Practice filtering in skill and AP styles.

Explore filtering Spark data frames to count flights that depart late at origin and arrive on time, using filter and where clauses, with Spark SQL style in Scala.

Filter Spark data frames to count flights with departure delay over 60 minutes, using sql-style and api-style methods, and preview results in the airlines dataframe (count = 40,104).

Filter spark data frames to count flights that depart on time or early and arrive at least 15 minutes late, using boolean and with Spark SQL and Scala.

Filter spark dataframes with the in operator in scala to count flights from five major US airports, and validate the results with distinct data.

Derive a flight date by padding month and day to two digits, then filter Spark data frames with between or like operators to count delayed flights from 2008-01-01 to 2008-01-09.

Count flights departing late on Sundays from the 2008 January data using Spark data frames in scala. Use date_format and to_date to extract day of week and filter late departures.

Learn to perform aggregations on Spark dataframes by grouping by a key and applying count, sum, min, max, and aliasing with agg.

Learn how to sort data frames in Spark using data frame APIs, with ascending and descending options, and see a practical flight count example.

Learn to compute the total flight count and counts of departure- and arrival-delayed flights using Spark DataFrame APIs in Scala. Combine filtering and aggregation into a single-row dataframe.

Build a single dataframe with one row containing flight count, departure delayed count, and arrival delayed count using Spark data frame APIs, case when, sum, and alias.

Compute daily flight counts and delay counts (departure and arrival) using Spark DataFrame APIs, producing columns: flight date, flight count, departure delayed count, arrival delayed count, sorted by flight date.