Real-World Data Science with Spark 2

Gain an overview of Apache Hadoop, its distributed file system and map reduce framework. Track MRV1 to MRV2 evolution and how YARN unifies processing on HDFS.

Explore Apache Spark and its in-memory processing advantages over Hadoop MapReduce, and learn Spark libraries, the DAG execution model, and multi-language support across cluster managers.

Install Apache Spark on your laptop from tarball or using Docker and run Spark jobs locally. Then learn to scale to the cloud with Azure, using a two-setup workflow.

Install and configure Apache Zeppelin, a web-based notebook for Spark with Matplotlib and ggplot2, using binary packages, a Docker image, or from source, to access Spark, R, and Python APIs.

Explore the structure of data, from RDBMS data types to HDFS, and learn how Spark SQL and DataFrame API enable distributed, schema-aware processing across diverse data sources.

Explore the core rdd api by reading from disk, filtering and transforming data, and writing results back to disk using map, mapPartitions with index, and repartition.

Learn to work with data in Spark using dataframe, dataset, and SQL by loading data, creating schemas, registering tables, and performing transformations and visualizations.

Learn to manipulate rows and columns of Spark data frames by merging frames with union, filtering, sorting, grouping, and aggregating, then adding or renaming columns for feature engineering.

Learn how to manage different file formats in Spark data processing, including csv to json conversion, reading with type inference options, selecting columns, filtering, and writing new files.

Visualize data with ggplot2, matplotlib, and angular.js in Zipline notebooks, using inline plots and interactive displays. Utilize Zeppelin display to build interactive forms and capture user input to run code.

Explore spark.ml and spark.mllib, the out-of-the-box libraries for classification, regression, clustering, and pipelines. Train models with svm, split data for training and testing, and evaluate with roc auc.

Wraps up basic statistics and linear algebra in Spark, covering random data generation, descriptive statistics, correlations, cross tabulation, and vector and matrix operations for distributed data.

Preprocess data by cleansing, parsing text, handling missing and formatting issues, and joining sources, then extract and transform features to build robust data pipelines for analytics with Spark.

Explore dimensionality reduction with PCA, projecting correlated features into uncorrelated principal components to obtain a lower-dimensional representation and interpret explained variance.

Build a multi-stage Spark ml pipeline that combines a tokenizer, term frequency, and logistic regression to train, transform, and save reusable models on a data frame with labeled documents.

Learn how to set up a Twitter app, obtain consumer keys and access tokens, and stream tweets using Spark streaming to collect and save them to disk in micro-batches.

Stream tweets in real time, bind a Twitter stream to a Leaflet map using Zipline and Scala, and display geolocated tweets as they stream.

Learn how to cleanse and build a reference dataset by loading and merging tweets, detecting language, and saving to parquet format for scalable analysis.

Explore a cleansed Twitter reference dataset with Escorial to query and visualize tweets using SQL, perform exploratory data analysis, and examine text length, timestamps, geo locations, and followers.

Explore basic indicators and correlations using a statistics object, learn stratified sampling for real data, and compare exact versus approximate statistics with Spark.

Assess statistical relevance in real-world datasets using Spark with tests like t-square, goodness-of-fit, independence tests, Kolmogorov–Smirnov, and kernel density estimation to compute p-values.

Apply singular value decomposition and principal component analysis to a tweet feature matrix to reduce features for downstream modeling.

Extend basic statistics to your needs by adding an operator to an existing RTD or by extending the RTG class, using extension classes and implicit conversions for custom KPI calculations.

Learn to convert tweets into numerical feature vectors using bag of words, tokenization, stop word removal, and tf-idf, enabling text data to feed machine learning models.

Discover how to customize text feature extraction for Twitter data with stemming, syntax handling, lemmatization, and a Twitter model. Build a tokenization and analysis pipeline to handle hashtags and mentions.

Detect tweet sentiment using the Stanford NLP library with a token-based weighting from very positive to very negative, zero for neutral. Explore integrating lexicons to improve accuracy.

Identify topics in tweets using lda, build a vocabulary, convert text to feature vectors, and fit an lda model to map topics to terms.

Explore creating word clouds from text data by calculating word counts, selecting top words, and rendering visuals with the R word cloud library, Python, and Matlab's matplotlib.

Locate users and aggregate their locations into heatmaps using GeoHash encoding. Create a geocache and render the map with the leaflet library to visualize tweets around London.

Collaborate on the same note with peers using web socket, authenticate users, and set note permissions in zipline. Explore interpreter binding options: shared, scoped, and orated for collaborative work.

Create visual dashboards and share live, interactive insights with business stakeholders by embedding widgets, using skins, and updating dashboards in real time via web sockets.