Cloudera CCA 175 Spark Developer Certification: Hadoop Based

Explore how to use the map function in Spark to transform raw data into keyed records, create compound keys, handle delimiters, and build structured sales records for analytics.

Compute profit margin by aggregating total sales and profits, then determine average sales per customer, average profit per sale, and the role of discounts using map, reduce, and distinct.

Apply reduce and reduceByKey to aggregate flight data in Spark, computing total miles, total flights, and average miles per flight, then count departures by airport using a map and sort.

Explore reduceByKey with compound values to compute average airline delays by summing delays and counts, then dividing. Handle zero delays to avoid not a number and present results.

Explore a Spark reduceByKey aggregation on compound keys of day of week and time of day, mapping delay values and sorting by the key for clear output.

Explore Spark transformations and actions on RDDs, including map, filter, distinct, and reduce by key, then collect results.

Configure spark resources via command line to handle large datasets, adjusting memory and cores with yarn and spark shell for efficient counts on a 500 MB flight data set.

Learn to compute delay chances with aggregations and groupByKey, building a flights map by airline and calculating delay percentages at chosen thresholds.

Explore loading data into a Spark dataframe, map and filter fields, and perform sql-like operations with order by column name, select, and simple aggregations.

Learn to perform dataframe aggregations by neighborhood, computing average price, max price, and counts; sort results; and apply user defined functions to shape and convert data into an RTT.

Explore hive string manipulation using concat and concat_ws, along with lower, upper, case, length, substr, trim, and split to format and combine customer data.

Explore string functions such as replace, lpad, unstring, in string, lower, and like to format, search, and clean data, including replacing domains and aligning emails in tabular views.

Learn how to compute date differences with date_diff, add months and days with date_add, and format dates with date_format, then group rentals by day of the week to count occurrences.

Learn to use hive conditional statements, including if and case, to classify data as on time or late, and to handle nulls with is null and is not null.

Master aggregations in spark by computing global and per-item averages, such as average payments by customers, average rentals per CD, and actor film earnings using group by and joins.

Explore multi-column ordering in Hive using order by, distribute by, sort by, and cluster by to control sorting and parallelization. Learn global versus local sorting across partitions with practical examples.

Explore Sqoop meta information commands to inspect databases, tables, and sample data without logging in, using list databases, list tables, eval, and help for import control arguments.

Explore how parallel processing works in Hadoop-based imports using mappers, split by columns, and primary key considerations.

learn to import all tables from a database with parallelization, manage primary key absence by using a single mapper for non-key tables, and validate with a copy table.

Explore storing data in four major file formats - avro, simple binary/sequence file, text, and parkway file format - and learn how columnar parkway enables fast aggregations and efficient data loading with Spark shell.

Demonstrates advanced Hive import techniques, including overwriting data, importing all tables, and using high control arguments, with date casting and result verification.

Recaps how to leverage the Spark framework to transform data, derive analysis and insights from big data, and prepare for the Cloudera CCA 175 Spark Developer Certification.