Apache Spark 2 and 3 using Python 3 (Formerly CCA 175)
What you'll learn
- All the HDFS Commands that are relevant to validate files and folders in HDFS.
- Python Fundamentals to write the required code while solving problems in the scenario based exam
- Ability to use Spark SQL to solve the problems using SQL style syntax.
- Pyspark Dataframe APIs to solve the problems using Dataframe style APIs.
- Relevance of Spark Metastore to convert Dataframs into Temporary Views so that one can process data in Dataframes using Spark SQL.
Requirements
- Basic programming skills using any programming language
- Self support lab (Instructions provided) or ITVersity lab at additional cost for appropriate environment.
- Minimum memory required based on the environment you are using with 64 bit operating system
- 4 GB RAM with access to proper clusters or 16 GB RAM with virtual machines such as Cloudera QuickStart VM
Description
As part of this course, you will learn all the key skills to build Data Engineering Pipelines using Spark SQL and Data Frame APIs using Python as Programming language. This course used to be CCA 175 Spark and Hadoop Developer course for the preparation of Certification Exam. As of 10/31/2021, the exam is sunset and we have renamed it to Apache Spark 2 and 3 using Python 3 as it covers industry relevant topics beyond the scope of certification.
About Data Engineering
Data Engineering is nothing but processing the data depending upon our downstream needs. We need to build different pipelines such as Batch Pipelines, Streaming Pipelines, etc as part of Data Engineering. All roles related to Data Processing are consolidated under Data Engineering. Conventionally, they are known as ETL Development, Data Warehouse Development, etc.
Course Details
Here is the high level outline of the topics related to this course.
Quick recap of Python
Data Engineering using Spark SQL
Let us, deep-dive into Spark SQL to understand how it can be used to build Data Engineering Pipelines. Spark with SQL will provide us the ability to leverage distributed computing capabilities of Spark coupled with easy-to-use developer-friendly SQL-style syntax.
Getting Started with Spark SQL
Basic Transformations
Managing Tables - Basic DDL and DML
Managing Tables - DML and Partitioning
Overview of Spark SQL Functions
Windowing Functions
Data Engineering using Spark Data Frame APIs
Spark Data Frame APIs are an alternative way of building Data Engineering applications at scale leveraging distributed computing capabilities of Spark. Data Engineers from application development backgrounds might prefer Data Frame APIs over Spark SQL to build Data Engineering applications.
Data Processing Overview
Processing Column Data
Basic Transformations - Filtering, Aggregations, and Sorting
Joining Data Sets
Windowing Functions - Aggregations, Ranking, and Analytic Functions
Spark Metastore Databases and TablesPlease note that the syllabus is recently changed and now the exam is primarily focused on Spark Data Frames and/or Spark SQL.
All the demos are given on our state of the art Big Data cluster. You can avail one-month complimentary lab access by reaching out to support@itversity.com with Udemy receipt.
Who this course is for:
- Any IT aspirant/professional willing to learn Big Data and give CCA 175 certification
- Python Developers who want to learn Spark to add the key skill to be a Data Engineer
Featured review
![Kushwanth Chowdary](data:image/svg+xml,%3Csvg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64"%3E%3C/svg%3E){kind=avatar}
Course content
- Preview08:01CCA 175 Spark and Hadoop Developer - Curriculum
- 02:25Setup Development Environment (Windows 10) - Introduction
- 04:12Setup Development Environment - Python and Spark - Pre-requisites
- 03:07Setup Development Environment - Python Setup on Windows
- 02:31Setup Development Environment - Configure Environment Variables
- 05:28Setup Development Environment - Setup PyCharm for developing Python applications
- 02:31Setup Development Environment - Pass run time arguments or parameters
- 01:38Setup Development Environment - Download Spark compressed tar ball
- 01:00Setup Development Environment - Install 7z for uncompress and untar on windows
- 02:26Setup Development Environment - Setup Spark
- 06:05Setup Development Environment - Install JDK
- 03:46Setup Development Environment - Configure environment variables for Spark
- 06:30Setup Development Environment - Install WinUtils - integrate Windows and HDFS
- 07:06Setup Development Environment - Integrate PyCharm and Spark on Windows 10
Instructors
- 4.4 Instructor Rating
- 8,600 Reviews
- 218,469 Students
- 15 Courses
13+ years of experience in executing complex projects using vast array of technologies including Big Data and Cloud.
ITVersity, Inc. - a US based organization to provide quality training for IT professionals and we have the track record of training hundreds of thousands of professionals globally.
Building IT career for people with required tools such as high quality material, labs, live support etc to upskill and cross skill is paramount for our organization.
At this time our training offerings are focused on following areas:
* Application Development using Python and SQL
* Big Data and Business Intelligence
* Cloud
* Datawarehousing, Databases
- 4.3 Instructor Rating
- 4,639 Reviews
- 173,021 Students
- 7 Courses
We have built a team to support going forward. If you send messages to this account for our courses, they will be sent to our Helpdesk from where we will be rewriting to our team.