
Learn to harness Python with Apache Spark for big data using PySpark, covering Spark SQL and data frames, MLlib, streaming, plus hands-on labs, mock projects, and online forums.
Let's explain the set-up for the course!
Set up a Databricks account on AWS, launch a six gigabyte community edition cluster, upload data from local or S3, and work with a data frame via Spark SQL.
Let's walk through the local installation of Ubuntu
Install findspark with pip3, initialize it with your Spark home path, and import pyspark from any directory or Jupyter notebook to run PySpark.
Let's show you how to use Amazon Web Services' EC2 Instances for Spark!
Create an EC2 instance in aws console, then connect via SSH from Windows, Mac, or Linux to set up spark and jupyter on Ubuntu, including key pairs and PuTTY usage.
Connect to your EC2 instance from a Mac or Linux terminal using a private .pem key, secure it with chmod 400, and ssh to the instance's DNS address.
Explore a quick Python crash course for Spark and PySpark. Review the Jupyter notebook approach and flexible IDE options, with optional exercises.
learn solutions to the Python crash course exercises, including exponentiation, string operations, and accessing nested lists and dictionaries, plus domain_get, count_dog, and a birthday-adjusted speeding check.
Learn the basics of Spark DataFrames!
Learn some basic operations with Spark 2.0
Learn to group Spark DataFrames by a column, apply aggregate functions (mean, sum, max, min, count) via groupBy or agg, and use aliasing and order by to present clean results.
Explore how to handle missing data in Spark with PySpark: drop or fill nulls, use thresholds and subset filters, and compute mean imputation for numeric columns.
Load walmart_stock.csv with a spark session, header, and infer schema, inspect columns, then compute HV ratio, descriptive stats, mean, max and min, and monthly averages.
Explore machine learning with Spark and MLlib using the Spark 2.0 data frame syntax. Learn to format data into a features column and build pipelines with vector indexers in PySpark.
Learn the latest Big Data Technology - Spark! And learn to use it with one of the most popular programming languages, Python!
One of the most valuable technology skills is the ability to analyze huge data sets, and this course is specifically designed to bring you up to speed on one of the best technologies for this task, Apache Spark! The top technology companies like Google, Facebook, Netflix, Airbnb, Amazon, NASA, and more are all using Spark to solve their big data problems!
Spark can perform up to 100x faster than Hadoop MapReduce, which has caused an explosion in demand for this skill! Because the Spark 2.0 DataFrame framework is so new, you now have the ability to quickly become one of the most knowledgeable people in the job market!
This course will teach the basics with a crash course in Python, continuing on to learning how to use Spark DataFrames with the latest Spark 2.0 syntax! Once we've done that we'll go through how to use the MLlib Machine Library with the DataFrame syntax and Spark. All along the way you'll have exercises and Mock Consulting Projects that put you right into a real world situation where you need to use your new skills to solve a real problem!
We also cover the latest Spark Technologies, like Spark SQL, Spark Streaming, and advanced models like Gradient Boosted Trees! After you complete this course you will feel comfortable putting Spark and PySpark on your resume! This course also has a full 30 day money back guarantee and comes with a LinkedIn Certificate of Completion!
If you're ready to jump into the world of Python, Spark, and Big Data, this is the course for you!