AWS & Apache Spark: The Complete Big Data Analytics Course

Set up a Spark-ready AWS environment by creating a VPC and launching a basic EC2. Select a t2 micro, enable a public IP, and stop the instance to avoid charges.

Sign in to the AWS console, start the EC2 instance, and connect via ssh using the pem file after setting its permissions, noting DNS name changes after restart.

Connect to your AWS instance using PuTTY on Windows, or SSH on Linux and Mac, generate a PPK with PuTTYgen, and configure a saved Spark session.

Log in to the AWS console and launch an Amazon Linux EC2 instance with a public IP, then install Spark 2.0 and clone the Amplab Spark EC2 branch 2.0.

Launch aws spark clusters using spark ec2 scripts to create a master and a slave. SSH into the master to access the spark master page and start a spark session.

Terminate your clusters at chapter end to avoid surprises and costs; restart clusters when you resume, waiting about fifteen minutes for them to fire up.

Set up Spark clusters on AWS, launch and connect to the master, and run SparkR with RStudio to create and manipulate Spark dataframes using the faithful dataset.

Explore gaussian generalized linear models in spark using iris and diabetes data, fitting linear and binomial models, and evaluating with rmse and model summaries.

Apply binomial generalized linear models to the titanic dataset, convert text to dummy features, split data, and evaluate survival predictions with a 0.5 cutoff achieving about 82.07% accuracy in Spark.

Conclude the chapter by introducing naive Bayes and K-means in Spark ML, illustrating supervised and unsupervised modeling with Titanic and iris data, and show cluster destruction on EC2.

Explore big data workflows with S3, Spark, SQL, and Dplyr by uploading large datasets to S3 and running scalable analytics on a Spark cluster.

Explore handling large data frames with a dplyr-like syntax, read from s3 into a spark dataframe, and ingest csv data directly for scalable analytics.

Explore SparkR dataframes with dplyr-like commands such as select, filter, and summarize, and learn group by, aggregate, mean, and piping with the Margaret R package.

Master spark sql to query spark dataframes directly via a temporary view like diabetes_spark_table. Learn to use select, distinct, and count with proper quoting to analyze data across a cluster.

Explore hdfs, the Hadoop distributed file system, which stores data in blocks across commodity machines for scalable, redundant, high-bandwidth access and easy integration with Spark.

Explore Databricks community edition to manage Spark clusters with interactive notebooks, six gigabytes of free space, and multi-language support for Python, Scala, SQL, and R.

Review SparkR concepts, AWS EC2 cluster setup, and cluster management; summarize data reading, gaussian and binomial modeling, and SparkR with S3 and HDFS.