Big-Data with PySpark, Spark Streaming, Spark ML and Kafka

Install Anaconda with Python 3.7 64-bit, set up a Python notebook environment, install Java runtime, Spark and Hadoop binaries, configure environment variables, and run Spark from the Windows command prompt.

Install Anaconda on Windows by downloading Python 3.7, running the installer, and adding Anaconda to environment variables. Verify the installation from the command prompt.

Install the Java runtime environment on Windows by downloading from java.com and running the setup. Verify installation by opening the command prompt and typing java -version.

Download pre-built Spark 2.3.3 with Hadoop 2.6 binaries, extract the archive with 7-zip, and set environment variables to start Spark and the console.

Install and configure cloud storage integration for Azure, AWS, and GCP by adding Hadoop and storage connector jars to Spark and setting up access keys.

Explore cloud storage installation and setup across Azure, AWS, and GCP, creating buckets and credentials for seamless Spark Hadoop access.

Learn to set up a Spark context in Python, initialize a local Spark context, and perform parallelized RDD operations such as take and reduce.

Learn how to access cloud storage with Spark, reading text files from S3, Azure blob storage, and GCP, using textFile and take five, with core-site and library configuration.

Compare spark context and spark session as the single entry point to spark functionality, and learn how spark session unifies access to data frames, Hive, and streaming contexts for lessons.

Explore spark data frames, a distributed, named-column table-like structure akin to a relational table, with catalyst optimization, supporting json, parquet, hive formats, s3 reads, and seamless MLlib integration.

Explore RDD transformations and actions in Spark, focusing on the 14 useful RDD methods, and learn how lazy transformations trigger computations via map, take, count, filter, flatMap, and distinct.

Explore advanced rdd transformations and actions, including sampling with or without replacement using a seed, extracting keys and values, joining and zipping rdds, and reducing by key to aggregate results.

Explore more than 17 Spark data frame methods to transform and act on a data frame, including read, show, count, print schema, cache, drop duplicates, and dropna with configurable parameters.

Master dataframe transformations and actions in spark with select, distinct, fill, filter, describe, summary, group by, and order by; register as a table, convert to pandas, and write to storage.

Master RDD operations on an Azure HD Insights cluster, performing map, flat map, filter, union, intersection, distinct, group by key, sort by key, and reduce by key, plus data frames.

Execute data frame operations on a clustered Azure HD Insights environment using PySpark, including counting, describing, dropping duplicates, registering temp tables, visualizing, and saving results to Azure storage.

Set up an AWS EMR cluster with Spark, create a key pair, launch a notebook, and run PySpark via Pye's Park to create a Spark session.

Learn big data ingestion and preprocessing with Spark dataframes by loading NYC taxi trip and fare data, performing left joins on medallion, dropping columns, and multi-criteria filtering.

Learn how to use PySpark SQL to query data frames: create a Spark session, register temp views, and apply select, where, count, distinct, order by, aliases, and cross join.

Explore Spark Streaming and Structured Streaming with Kafka for real-time IoT architectures, enabling aggregation, filtering, and correlation on streaming data. Contrast batch and stream processing, noting windows, latency, and insights.

Explore popular big data streaming frameworks, including Kafka as a streaming bus pipeline, Storm for real-time processing, and Spark Streaming and Structured Streaming.

Compare Spark Streaming's micro-batch real-time processing with Structured Streaming's event-time aware data frames, triggers, and no batch concept, including late data handling and end-to-end guarantees.

Explore spark streaming hands-on with PySpark code, covering socket stream counts, file streams, and structured streaming, and build an IoT architecture using Kafka and Zookeeper.

Instantiate a Spark streaming context from the Spark context and read socket streams (localhost:9999) with a five-second batch to count words using flatMap, map, and reduceByKey, printing results.

Build a fault-tolerant big data IoT pipeline with PySpark, Kafka, and Zookeeper, streaming data via Spark Structured Streaming, performing etl on dataframes and publishing to topics.

Subscribe to a Kafka topic using Spark streaming, read data in Kafka format, cast keys and values to string, print to console, and manage offsets and checkpoints for fault-tolerant streaming.

Consume data from Kafka topic one with PySpark streaming, transform and push to topic two using a Kafka sink, with checkpointing, earliest offsets, and update mode.

Read from Kafka topic one, process with PySpark structured streaming, split words by spaces, and write the preprocessed data to Kafka topic two, using a defined schema.

Learn to stream a CSV file with spark readStream, define a schema, push data to Kafka, decode and convert JSON to a datastream table, and apply timestamped windowed aggregations.

Learn how Spark Streaming uses tumbling and sliding time windows to perform event-time aggregations, with five-minute windows, triggers, and late-data handling to update counts.

Implement tumbling and sliding windows in PySpark on Kafka timestamps, aggregating occupation counts, and explore window behaviors, including 1 minute tumbling, 10 minute sliding, and watermarking.

Learn how watermarking in Spark structured streaming handles late data to correctly update windowed aggregates, using tumbling windows and watermark delays with Spark and Kafka.

Explore Spark ML and MLlib, learning distributed training on Spark clusters, featuring regression, classification, clustering, and pipelines for scalable model building.

Create a Spark ML linear regression workflow to predict abalone age from physical measurements, encoding sex, assembling features, scaling, training, evaluating with RMS, and plotting the regression line.

Train a Spark ML logistic regression model on census data, using string indexers, one-hot encoding, and a pipeline to train, predict, and evaluate with area under ROC.

Demonstrate Spark ML decision trees by predicting heart disease from the Cleveland vitals, building a pipeline with vector assembler and a random forest classifier, and evaluating accuracy and auc.

Perform k-means clustering on the iris dataset with spark.ml, building a pipeline using vector assembler and standard scaler. Evaluate with silhouette score and view cluster centers.