Machine Learning with Apache Spark using Scala

Install openjdk 11 with apt update and apt install, confirm by running java -version to show 11.0.27, preparing the environment for Apache Zeppelin.

Set the java environment by editing /etc/profile with sudo nano, adding lines for java home, java path, and PATH, then verify the java 11 location with echo $JAVA_HOME.

Install apache zeppelin 0.12.0 on ubuntu by downloading, untarring, and starting the zeppelin daemon, then access localhost:8080 to create and manage notebooks.

Install docker desktop on windows 10/11 by downloading the installer, running it as administrator, and launching docker to access containers, images, volumes, and build features.

Learn to run Apache Zeppelin on Docker Desktop for Windows by pulling the 0.12.0 image, launching a Zeppelin container on port 8080, and starting, stopping, and viewing logs.

connect and configure apache spark with zeppelin by installing spark, starting spark shell locally, running a dataframe, and exporting spark home in bashrc for zeppelin integration.

Apache Zeppelin empowers big data analytics with a web-based, open-source notebook featuring multi-language support, real-time code execution, live visualizations, collaboration, and seamless integration with Spark, Hive, HDFS, and Hadoop.

Learn to format Apache Zeppelin notebooks with Markdown, including headers, emphasis, lists, links, images, tables, and quotes, and apply best practices for clear, collaborative reports.

Create, edit, and run paragraphs in Apache Zeppelin, using %Spark, %SQL, and %MD, to build modular data pipelines with code, markdown, and visual outputs.

Discover how Apache Zeppelin turns raw data into insights with tables and charts like bar, line, pie, and scatter plots, using SQL or Spark queries to build a sales dashboard.

Explore the five default chart types in zeppelin—bar, pie, area, line, and scatter—and see interactive visualizations generated from an employee dataframe and sample queries.

Explore how to configure and use the Apache Spark interpreter in Apache Zeppelin to run Spark code, visualize results, and perform SQL, DataFrame, and RDD operations.

Explore RDDs and DataFrames in Apache Spark within a Zeppelin notebook, performing RDD transformations, creating DataFrames, registering temporary tables, and visualizing results with SQL or charts.

Learn to run Spark SQL queries on DataFrames in Apache Zeppelin, register DataFrames as temporary views, and leverage in-memory caching to speed up interactive analysis and visualizations.

Explore how Apache Zeppelin visualizes Spark outputs by turning Spark data frames into interactive tables and charts using %SQL, with bar, line, and pie charts for meaningful insights.

Track Spark job performance and tune execution with Apache Zeppelin and the Spark web UI, exploring caching, repartitioning, and broadcasting joins to optimize data frames and pipelines.

Introduction to Spark, a high-performance general-purpose cluster engine that runs workloads across nodes and supports Java, Scala, Python, and R, with Spark SQL, MLlib, GraphX, and streaming.

Register for a free Databricks account by signing up with a work email on the community site, then confirm the email and sign in.

Create a free Databricks account using the community edition, obtain credentials, and log in to practice on the Databricks platform.

Log in to the website and go to clusters. Name your cluster and click create to provision a Spark cluster, then watch the status move from pending to active.

Learn the fundamentals of notebooks in the Databricks community: create and name notebooks, run runnable cells, and use magic commands like %python and %scala for coding and documentation.

Learn Apache Spark for in-memory performance and fast iterative algorithms, enable data sharing across jobs, and build complex analytics quickly with broad operators support and interactive shells.

Learn how Spark RDD and dataset models work in Scala, why datasets outperform RDD, and how Spark shifted after Spark 2.0, with practical creation and loading of resilient distributed datasets.

Load and explore a spark dataframe by creating an employee data frame, running sql queries, and visualizing results with bar charts and other plots.

Explore anonymous functions in Scala, defining unnamed functions with one, two, or no parameters, and use them in expressions like x plus one.

Explore working with Spark dataframes: load structured data, define a schema, select columns, filter rows, create a temporary in-memory object to query, use Spark SQL, and visualize results for modeling.

Explore spark dataframes with explicit schema and encoder, load railway data, derive schema, and compute descriptive stats and simple spark sql queries like average delay by day of week.

Explore Spark datasets, a typed interface that blends data frame and RDD, enabling efficient lambda-driven analysis in Scala and Java; create, load, and display datasets with filter and select.

Examine the two core machine learning approaches: supervised learning with regression and classification using labeled data, and unsupervised learning with unlabeled data, including k-means and principal component analysis.

Discover the typical machine learning process flow, emphasizing data preparation, cleansing, and analysis, and how training data quality and dataset size drive accurate predictions.

Explore Spark MLlib, the scalable machine learning library in Spark, covering classification, regression, clustering, collaborative filtering, and dimensionality reduction, plus data processing, pipelines, and hyperparameter tuning for exploratory data analysis.

Upload data by dropping files, locate the data file, and import via workspace to add the DBC file in the Data Brix environment.

Explore basic statistics and correlation, learning how to measure how two variables move together using Pearson and Spearman methods, and interpreting positive and negative correlations with scatterplots.

Learn how to use data sources to load data in Apache Spark 3.0 using Scala, and preview upcoming datasource concepts in the next lecture.

Learn how to load a CSV data source into a Spark data frame, infer or define a schema, handle header and malformed data, and query the data with temporary views.

Learn how to load, define schema, and query a JSON data source in Spark using Scala, including multi-line JSON handling, temporary views, and basic DataFrame operations.

Learn how to load a LIBSVM data source in Spark 3.0 with Scala, inspect label and feature records, and configure SVM options for a machine learning task.

Load and process image files in spark dataframes, drop invalid images, configure multiple-file locations, and print schema to support image recognition models for machine learning.

learn to import Avro data files, view their columns and date fields, and print the schema in Spark to understand how Avro data is loaded from the file store.

Load Parquet data into Spark using a data source and specify the file store location. Print the schema to view all columns and confirm loading from Parquet into Spark.

Explore how machine learning data pipelines load data from multiple sources, clean and extract features, train and validate models, and deploy them.

Explore a machine learning project example that illustrates a typical ML pipeline and the aim of predicting tomorrow's rain in Australia, as introduced for upcoming lectures.

Predict whether it will rain tomorrow in Australia from weather data using a Spark Scala machine learning pipeline with a user defined schema and a Spark DataFrame.

Transform string columns to numeric with a string indexer, assemble features with a vector assembler, split data 70/30, and train a logistic regression model to predict rain tomorrow in Australia.

Build and test a Spark machine learning pipeline, create 30% testing data with an assembler, engineer features, then predict and evaluate accuracy, precision, and recall.

Explore the components of a machine learning pipeline, including datasets and dataframes. Understand how estimators train models, parameters guide iterations, and predictions arise through pipeline stages.

Explore feature extraction, transformation, scaling, converting or modifying features, and selection from raw data, and learn how algorithms group these tasks into extracting, transforming, and selecting features.

Learn tf-idf feature extraction by tokenizing text, computing term frequency and inverse document frequency, and generating word vectors with a hashing-based transformer in Spark.

Apply Word2Vec as a feature extractor in Spark 3.0 with Scala to convert documents into vectors by averaging word vectors, enabling similarity calculations and predictions.

Explore CountVectorizer in Spark with Scala to convert text documents into numeric feature vectors, build a vocabulary, and transform data frames into feature columns.

Learn how the feature hasher converts multiple numerical and categorical features into a fixed-size vector using hashing tricks, with configurations for boolean and other column types.

Learn tokenization in Spark ML by using text organizers and regular expressions to split sentences into words, build frames, and apply a user defined function to count word length.

Learn to apply StopWordsRemover in Apache Spark 3.0 using Scala to filter out high-frequency stop words from text input, producing a filtered output column for downstream feature transformers.

Explore n-grams as a feature transformer in Spark, transforming word sequences into n-gram features and generating an output column via the transform function.

Learn how to binarize numerical features into binary values using a threshold, demonstrated with a spark data frame, input and output columns, and a 0.5 threshold example.

Learn how principal component analysis uses orthogonal transformation to convert correlated features into uncorrelated principal components, projecting data into a low-dimensional space with Spark's PCA transformer and feature columns.

Explore polynomial expansion to transform features into polynomial space, expanding two features to a degree-3 representation with the polynomial expansion class, applying to a data frame to create new features.

Master the discrete cosine transform (DCT) as a Spark data frame feature transformer: transform input columns to a new output column with DCT values, enabling comparison with the original data.

StringIndexer encodes string columns into numeric indices, converting each string label to a category index. It can index multiple columns and adds an output index column.

Learn how the index-to-string transformer recovers original labels from predicted indices by applying a string indexer and converting category indices back to the original strings in a data frame.

Demonstrate one-hot encoding of a categorical feature by mapping a two-column data frame to a vector with at most a single 1, then fit and transform to reveal category vector.

Explore sql transformer, a feature transformer in Spark ML, that applies sql-like expressions to a dataframe to create new features such as v3 and v4.

Explore how VectorAssembler, a Spark feature transformer, combines multiple input columns into a single output feature vector, with a hands-on example transforming ideas, mobile, and user feature columns.

Utilize RFormula as a feature selector in Spark ML: import the formula library, build a data frame, generate feature columns from country, and define a label to predict clicks.

Demonstrate ChiSqSelector for top feature selection on a labeled dataset, extracting the selected feature column from the feature data.

Explore how classification models use features to predict class labels, focusing on logistic regression and its probability via the logistic function for outcomes like on time or late.

Learn how a decision tree classifier classifies railway trains as on time or late using Spark and Scala, from data import and feature engineering to training, testing, and evaluating accuracy.

Train a logistic regression classifier on a railway delay dataset using Spark MLlib. Use day of week, origin, and departure delay as features, with vector assembler and a 70/30 split.

Train a naive bayes classifier on the iris dataset with Apache Spark and Scala, encoding labels, assembling features, and evaluating with a 70/30 split to achieve about 0.87 accuracy.

We build a random forest classifier project in Spark 3.0 using Scala to predict railway arrival delays, preparing data with vector assembler, training on a 70/30 split, and achieving 92% accuracy.

Train a gradient-boosted tree classifier to predict railway arrival delays using Spark 3.0 with Scala. Split data 70/30, assemble features, and evaluate accuracy at 92%.

Build a linear support vector machine classifier in spark 3.0 using scala, preparing numeric features with a vector assembler and a 70/30 train-test split, and evaluate with 87 percent accuracy.

Train a one-vs-rest classifier using logistic regression in Spark, preprocess a train arrival delay dataset with a vector assembler, and evaluate accuracy with a multiclass evaluator, achieving about 92 percent.

Explore the regression model as a predictive technique, examining the relation between dependent and independent variables to forecast outcomes, with a focus on linear regression.

Train a linear regression model on railway arrival delays using spark ml in Scala, selecting six numerical features, and evaluating with a 70/30 train‑test split to predict arrival times.

Explore building a decision tree regression model in Spark using Scala, from loading data and vector assembler to training, predicting, and evaluating with root mean square error.

Explore random forest regression in spark, building with vector assembler and train-test splits. Evaluate using a regression evaluator, yielding a root mean square error of 21 minutes.

Train a gradient-boosted tree regression model on Apache Spark 3.0 using Scala by loading railway delay data, preparing features, splitting 70/30, training, predicting, and evaluating with root mean square error.

Explore unsupervised learning with k-means clustering to segment mall customers by annual income and spending score, using vector assembler in Spark and Scala.

Learn random sampling with a 70/30 train-test split and how vector assembler creates a single feature vector for models like logistic regression. Explore transform operations and regularization to curb overfitting.

Predict ad clicks with a Spark Scala project by loading a csv, applying string indexing and vector assembly, training a logistic regression model on a 70/30 train-test split.

Thank learners for enrolling in the course and invite them to provide a different rating. Wish you all the best for your future.