Spark and Python for Big Data with PySpark

Set up a Databricks account on AWS, launch a six gigabyte community edition cluster, upload data from local or S3, and work with a data frame via Spark SQL.

Install findspark with pip3, initialize it with your Spark home path, and import pyspark from any directory or Jupyter notebook to run PySpark.

Create an EC2 instance in aws console, then connect via SSH from Windows, Mac, or Linux to set up spark and jupyter on Ubuntu, including key pairs and PuTTY usage.

Connect to your EC2 instance from a Mac or Linux terminal using a private .pem key, secure it with chmod 400, and ssh to the instance's DNS address.

Learn to install Spark and dependencies on EC2, set up Jupyter Notebook, connect Py4J and findspark, and securely access the notebook via the EC2 DNS in your browser.

Set up an emr cluster on aws with Zeppelin notebook to run spark and pyspark for big data analytics, and configure security groups, ssh access, and key pairs.

Explore a quick Python crash course for Spark and PySpark. Review the Jupyter notebook approach and flexible IDE options, with optional exercises.

Learn the basics of Jupyter Notebook for big data with PySpark, including .ipynb files, creating and running notebooks with cells, markdown notes, and Shift Enter output.

Master Python basics for big data with Spark and PySpark, covering integers and floats, strings, lists, dictionaries, booleans, tuples, sets, and core operations.

Explore defining and calling functions with def, parameters, and return versus print, plus lambda expressions, string methods, split, and list and dictionary operations.

learn solutions to the Python crash course exercises, including exponentiation, string operations, and accessing nested lists and dictionaries, plus domain_get, count_dog, and a birthday-adjusted speeding check.

Learn the Spark dataframe syntax in Python, with data stored in columns and rows, perform transformations, and show or collect results from a wide variety of sources.

Learn to work with Spark data frames by starting a Spark session, loading from JSON, inspecting the schema, and using select, with column, and with column renamed to transform data.

Learn to group Spark DataFrames by a column, apply aggregate functions (mean, sum, max, min, count) via groupBy or agg, and use aliasing and order by to present clean results.

Explore how to handle missing data in Spark with PySpark: drop or fill nulls, use thresholds and subset filters, and compute mean imputation for numeric columns.

Explore working with dates and timestamps in Spark dataframes using PySpark functions to extract year, month, day, and compute average closing prices by year from a stock dataset.

Load walmart_stock.csv with a spark session, header, and infer schema, inspect columns, then compute HV ratio, descriptive stats, mean, max and min, and monthly averages.

Explore machine learning with Spark and MLlib using the Spark 2.0 data frame syntax. Learn to format data into a features column and build pipelines with vector indexers in PySpark.