CCA 175 Exam preparation using Scala & Practice Tests [2020]

Explore Spark introduction and its in-memory, distributed data processing that’s faster than Hadoop, with multi-language support across Python, Scala, and Java. Focus on Spark SQL and DataFrames as core topics.

Explore resilient distributed datasets in Spark, an immutable, in-memory abstraction over data spread across multiple machines. Learn lazy transformations and actions that trigger actual execution on the distributed data.

Explore map and flatMap transformations in scala by applying functions to each element, calculating line lengths, and using flatMap to split lines into multiple output elements.

Apply the center transformation to filter elements by a predicate, such as even numbers or ages under 40, and compute the intersection to find common elements.

Learn how distinct transformation removes duplicates from an rdd, and how union transformation merges two rdds into one, with examples using in-memory data structures and concepts like map and flatMap.

Group data by key to cluster birthdays by month, triggering data reshuffling and a new stage, illustrated with Spark RDD transformations and the Spark Web UI.

Explore how reduceByKey aggregates total students per subject by shuffling data across a cluster, using an accumulator and x plus y to sum science, maths, and computer enrollments.

Sort data by key using Scala, ordering records by rule number in ascending or descending order with a sample file; also learn sorting by scores via transformation.

Learn how map partitions split data into partitions and enable per-partition processing with mapPartition and mapPartitionWithIndex, including partition index usage and parallel processing for RDDs.

Discover how to adjust partitions by repartition to increase or decrease counts; decreasing partitions avoids complete data reshuffling and can improve performance.

Explore joining datasets by key to combine names with email addresses, and compare inner, left outer, and right outer join operations.

Explore common spark actions such as collect, take, first, count, reduce by value, reduce, and for each, explain when to use them to manage memory usage and drive execution.

Explore spark data frames, a schema-based API faster than RDDs thanks to catalyst and tungsten. Learn how datasets blend the efficiency of data frames with type safety.

Learn how to read json files with Spark data frame API, handle multi-line json, merge schemas from multiple files, read from directories, and write json output.

Learn how parquet enables columnar storage with a built-in schema and snappy compression. Read, filter, and write data using Spark to parquet files, and inspect with parquet-tools.

Learn how to read CSV files in Spark as DataFrame, handle files with and without headers, apply a custom schema using struct fields, and write with compression options like gzip.

Read and write Avro files in Spark using the Avro package and a Spark shell. Learn how Avro stores schemas with data and uses Snappy compression for efficiency.

Master data frame column operations in Scala: add, rename, update, and drop columns, change data types, and derive new values with withColumn and substring.

Convert a string date column to a date type in a data frame, then extract year, month, day, and time components using date APIs; append date or timestamp as needed.

Master string manipulation with the data frame API: uppercase first names, lowercase last names, concatenate with a delimiter, replace patterns with regex, and split emails into parts.

Learn to filter data frame using filter api with predicates like email contains gmail.com, country in france or egypt, first name starts with d, and id between 1 and 10.

Learn to manipulate data frames with the API: select columns, add on-the-fly calculations, and sort by multiple keys; group by and aggregate by country.

Leverage dataframe API operations to filter data by country. Combine data frames with union, except, and intersect, and drop duplicates to reveal distinct countries.

Discover how to use Spark SQL by registering a dataframe as a temporary view and running SQL queries, including select of company, first name, and last name ordered by company.

explains how to access Hive tables from Spark, create a DataFrame, run Spark SQL on Hive, and write results back to Hive as a new table.

Explore Hive and SparkSQL string functions, including concat_ws for delimited joins, length, lower, upper, reverse, split, substring, and regexp_replace, demonstrated on the customers table.

Explore hive/sparksql date functions to extract month, year, day, and time components; add or subtract dates, compute date differences, work with unix timestamps, and format dates.

Explore hive/spark sql mathematical functions, including round, floor, rand, log, sqrt, power, factorial, sine, cosine, radians to degrees, and bit-shift operations, with practical demonstrations.

Explore Hive/Spark SQL analytics functions to group by city, count customers, and apply having and if logic, with MapReduce behind the scenes.

Demonstrate hive joins by combining the movies table with the movie rating table to show names and ratings. Explain inner and outer joins left, right, and full outer joins.

Explore Hive and SparkSQL window functions by comparing rank and dense_rank with category partitioning and price sorting, demonstrating how ties and skipping affect rankings across clothes, electronics, and home decor.

Explore exam-ready strategies with a simulated CCA 175 environment: practice timed questions, navigate an instructions screen, and learn flexible, output-based evaluation across languages.

Read snappy compressed input files with spark, select order ID and order status, and write the results as parquet using the specified compression codec to the output location.

Join the customer and orders CSVs to identify customers with more than four complete orders, using dataframe API and spark SQL, outputting first name and count sorted by count ascending.

Explore how to compute the maximum product price per product category using Spark SQL: group by category, max(price), and save as text with GSM compression ordered by maximum price descending.

Read a tab-delimited customer file and filter Lagos state residents using Spark. Define the output schema with customer_id, customer_name, and customer_city; cast id to integer and save with deflate compression.

Execute a spark-based transformation to extract the first three letters of the first name and save a tab-delimited, compressed file with customer ID, the first-name letters, and the last name.

Read a hive table in spark shell, filter Texas customers whose first name starts with t, and save the results as uncompressed parquet at the output location.

Learn to extract pending payment orders in March 2014 by loading a unix timestamp dataset into spark dataframes, filtering, counting by date, and exporting four json files.

Join customers and orders data, convert unix timestamps to a date, and filter 2013 complete orders; count orders per customer and write to a high table partitioned by state.