Spark Scala Real-World Coding Frameworks and Testing

Name: Spark Scala Real-World Coding Frameworks and Testing
Rating: 4.0 (270 reviews)

Spark Scala Framework, Hive, IntelliJ, Maven, Logging, Exception Handling, log4j, ScalaTest, JUnit

Created byFutureX Skills

Last updated 9/2026

English

English [Auto],

What you'll learn

Spark Scala industry standard coding practices - Logging, Exception Handling, Reading from Configuration File
Unit Testing Spark Scala using JUnit , ScalaTest, FlatSpec & Assertion
Building a data pipeline using Hive, Spark and PostgreSQL
Spark Scala development with Intellij, Maven
Cloudera QuickStart VM setup on GCP

Course content

7 sections • 34 lectures • 3h 45m total length

Introduction1:29
Discover the Spark Scala coding framework and testing, with hands-on unit testing using JUnit and Scala test, building a production ready data pipeline that interfaces with Hadoop, Hive, and PostgreSQL.
What is Big Data Spark?2:24
Discover distributed storage and computing with Spark, and compare it to MapReduce. Learn to build Spark Scala applications using DataFrames, Spark SQL, and Hive on HDFS and cloud storage.

Installing JDK, IntelliJ, and Winutils for Spark, Scala, and Hive Programming on29:47
Set up a Windows environment for Spark, Scala, and Hive with IntelliJ. Learn to install JDK 11, IntelliJ, winutils, and configure Spark core and SQL dependencies for a Maven project.
For Mac users - Installing JDK and IntelliJ and Spark Scala Hive Hello World16:39
Following Along on Windows and Mac0:36
Master IntelliJ setups on Windows and Mac. Learn that older Java and Scala code runs on newer versions without issues, keeping concepts and code consistent.
Scala Basics12:42
Master the basics of Scala by creating a Maven project, configuring Scala, and learning var versus val, type inference, arrays, lists, maps, tuples, and loops.
Installing PostgreSQL3:55
Fetching PostgresSQL data to a Spark DataFrame4:43
Connect a Spark application to PostgreSQL via JDBC, configure user credentials and URL, and fetch data from the future schema's Future X Course Catalog table into a DataFrame.
Importing a project into IntelliJ4:39
Import a spark scala project into IntelliJ from existing sources, configure maven with pom.xml, set JDK 1.8 and scala 2.11.8, and resolve dependencies.

Organizing code with Objects and Methods9:43
Organize a Spark Scala project by creating a common package and objects to initialize the Spark session and centralize PostgreSQL properties, enabling methods to fetch data frames from PostgreSQL tables.
Implementing Log4j SLf4j Logging5:32
Configure log4j2 and SLF4J logging in a Maven-based Scala app with a Log4j2 properties file. Replace println with logger.info and experiment with info, warn, and error levels for structured logs.
Exception Handling with try, catch, Option, Some and None6:20

Reading from Hive and Writing to Postgres10:47
Read data from a Hive table with Spark SQL, apply transformations to replace nulls with unknown or zero, and write the results to a PostgreSQL table.
Reading Configuration from JSON using Typesafe11:14
Learn how to read json configuration with typesafe config, fetch the Postgres target table from a json file, and control data transformations via json-driven settings.
Reading command-line arguments and debugging in InjtelliJ8:40
Learn how to pass and read command line arguments in a Spark program, debug with breakpoints in IntelliJ, and control program flow based on environment like dev or production.
Writing data to a Hive Table4:34
Managing input parameters using a Scala Case Class4:25

Scala Unit Testing using JUnit & ScalaTest6:47
Spark Transformation unit testing using ScalaTest12:28
ScalaTest - Testing Errors, Using Matchers, and Sharing Fixtures7:04
Explore unit testing in scala using spark transformer tests, catching null pointer exceptions, asserting results, and leveraging fixtures and matchers to validate data frame transformations.
Throwing Custom Error and Intercepting Error Message2:35
Throw and catch custom exceptions in Scala, intercept error messages, and validate data with tests that trigger a null course name in a Spark Scala workflow.

PySpark coding framework course preview1:40
PySpark course link with coupon
https://www.udemy.com/course/pyspark-python-spark-hadoop-coding-framework-testing/?referralCode=F624A76C9626B8884BE3
Congratulations & Thank You1:08
Thank you for enrolling in the Spark Scala course; rate it if you found it valuable, and explore resources, exclusive coupons, YouTube content, and educational blogs for further learning.

Big Data concepts6:05
Hadoop concepts1:44
Explore how Hadoop enables distributed storage and computing through hdfs and mapreduce, with data nodes and a name node coordinating a scalable cluster.
Hadoop Distributed File System (HDFS)1:59
Explore how HDFS distributes large data sets across a cluster using 128 MB blocks and a replication factor of three for fault tolerance, with trade-offs between storage and resilience.
Understanding Google Cloud (GCP) Dataproc1:27
Signing up for a Google Cloud free trial1:44
Sign up for a Google Cloud free tier account and explore data engineering and machine learning with $300 in free credits, then monitor usage in the billing console.
Creating a Dataproc Cluster3:34
Create a production-grade Hadoop and Spark cluster on Google Cloud Dataproc by configuring a three-node setup, enabling the Dataproc API, and SSH into the master node.
Storing a file in HDFS9:45
Learn to store and manage files in HDFS using Hadoop fs commands on a Dataproc cluster, including creating directories, putting files, listing, getting, and removing files.
MapReduce and YARN4:21
Hive1:29
Discover Hive, a SQL-like tool that processes data in HDFS by translating queries to MapReduce. Learn how Hive stores metadata in metastore and uses tables and Hiveql to query data.
Querying HDFS data using Hive11:43
Query HDFS data with Hive on a Dataproc cluster. Map CSV data to Hive tables, both internal and external, and run MapReduce jobs via Yarn.
Analyzing a billion records with Hive12:14

Requirements

Basic programming skills
Basic database skills
Hadoop entry level knowledge

Description

This course bridges the gap between academic learning and real-world application, preparing you for an entry-level Big Data Spark Scala Developer role. You'll gain hands-on experience with industry best practices, essential tools, and frameworks used in Spark development.

What You’ll Learn:

Spark Scala Coding Best Practices – Write clean, efficient, and maintainable code
Logging – Implement logging using Log4j and SLF4J for debugging and monitoring
Exception Handling – Learn best practices to handle errors and ensure application stability
Configuration Management – Use Typesafe Config for managing application settings
Development Setup – Work with IntelliJ and Maven for efficient Spark development
Local Hadoop Hive Environment – Simulate a real-world big data setup on your machine
PostgreSQL Integration – Read and write data to a PostgreSQL database using Spark
Unit Testing – Test Spark Scala applications using JUnit, ScalaTest, FlatSpec & Assertions
Building Data Pipelines – Integrate Hadoop, Spark, and PostgreSQL for end-to-end workflows
Bonus – Set up Cloudera QuickStart VM on Google Cloud Platform (GCP) for hands-on practice

Prerequisites:

Basic programming knowledge
Familiarity with databases
Introductory knowledge of Big Data & Spark

This course provides practical, hands-on training to help you build and deploy real-world Spark Scala applications. By the end of this course, you’ll have the confidence and skills to build, test, and deploy Spark Scala applications in a real-world big data environment.

This course uses high-quality AI-generated text-to-speech narration to complement the powerful visuals and enhance your learning experience.

Who this course is for:

Students looking at moving from Big Data Spark academic background to a real world developer role

Spark Scala Real-World Coding Frameworks and Testing

What you'll learn

Explore related topics

Course content

Introduction2 lectures • 4min

Environment Setup & Spark Scala basics7 lectures • 1hr 13min

Coding Best Practices3 lectures • 22min

A Data Pipeline with Hive, Spark and Postgres5 lectures • 40min

Spark Scala Unit Testing using ScalaTest4 lectures • 29min

Where to go from here?2 lectures • 3min

Appendix - Big Data Hadoop Hive for beginners11 lectures • 56min

Requirements

Description

Who this course is for: