
Master staging data with map operations to prepare it for transformations, then transform using Spark and Spark data frames, and explore storing data with Apache scoop.
Please download the data we will be using in this course below
Explore how to use the map function in Spark to transform raw data into keyed records, create compound keys, handle delimiters, and build structured sales records for analytics.
Explore reduceByKey with compound values to compute average airline delays by summing delays and counts, then dividing. Handle zero delays to avoid not a number and present results.
Explore Spark transformations and actions on RDDs, including map, filter, distinct, and reduce by key, then collect results.
Configure spark resources via command line to handle large datasets, adjusting memory and cores with yarn and spark shell for efficient counts on a 500 MB flight data set.
Explore hive string manipulation using concat and concat_ws, along with lower, upper, case, length, substr, trim, and split to format and combine customer data.
Master aggregations in spark by computing global and per-item averages, such as average payments by customers, average rentals per CD, and actor film earnings using group by and joins.
Explore multi-column ordering in Hive using order by, distribute by, sort by, and cluster by to control sorting and parallelization. Learn global versus local sorting across partitions with practical examples.
Explore how parallel processing works in Hadoop-based imports using mappers, split by columns, and primary key considerations.
Demonstrates advanced Hive import techniques, including overwriting data, importing all tables, and using high control arguments, with date casting and result verification.
Recaps how to leverage the Spark framework to transform data, derive analysis and insights from big data, and prepare for the Cloudera CCA 175 Spark Developer Certification.
Apache Spark is the single most revolutionizing phenomenon in Big Data Technologies. Spark turns infrastructure into a service, making provisioning hardware fast, simple, and reliable. Knowing this, many companies are transporting their big data analysis, staging, and storing needs to the Spark Framework. In this course, I will be preparing you for the CCA 175 Spark Developer Certification. This is the most popular and a very potent certificate in the Big Data realm.
In order for you to be able to get into this new realm of intense Tech competition, you will need a course to guide your way in Spark. The problem is that most courses are not designed to help you learn by example (immersion is the most potent way of learning in humans). Rather they bathe you with inapplicable information that you have to learn over and over again anyways.
This course is designed to cover the end-to-end implementation of the major components of Spark. I will be giving you hands on experience and insight into how big data processing works and how it is applied in the real world. We will explore Spark RDDs, which are the most dynamic way of working with your data. They allow you to write powerful code in a matter of minutes and accomplish whatever tasks that might be required of you. They, like DataFrames, leverage the Spark Lazy Evaluation and Directed Acyclic Graphs (DAG) to give you 100x better functionality than MapReduce while writing less than a tenth of the code. You can execute all the Joins, Aggregations,Transformations and even Machine Learning you want on top of Spark RDDs. We will explore these in depth in the course and I will equip you with all the tools necessary to do anything you want with your data.
I have made sure that this journey becomes a fun and learning experience for you as the student. I have structured this course so that you can learn step by step how Spark works and you can do the activities that I do in the course yourself. As you do these activities, you will become a master of Spark and complete any exercise asked of you on the CCA 175 certification exam.
There is no risk for you as a student in this course. I have put together a course that is not only worth your money, but also worth your time. I urge you to join me on this journey to learn how to dominate the IT world with the one of the most popular Big Data Processing Frameworks: Apache Spark.