Machine Learning: Data Analysis 2017

Explore how Hadoop's distributed storage and execution layers enable scalable data processing with map and reduce, featuring a distributed file system, data replication, and patterns like filtering and word count.

Explore how parallel dataflow systems enable scalable batch and real-time analytics, from map and reduce basics to streaming and micro-batch processing with lambda architecture and Spark-like engines.

Explore how Spark handles data flow with transformations and actions, collecting results to the driver, and using caching, partitions, and key-value operations.

Explore vectors and matrices in spark by performing in-memory calculations, converting between spark and breeze, and implementing least squares regression with jvm tools and sbt.

Explore spark's machine learning libraries sparc and elop, load lip svm data, and train logistic regression with lbfgs, then evaluate using binary classification metrics and roc auc.

Explore Spark machine learning workflows using data frames, transformers, and estimators to build pipelines that preprocess data, extract features, and train models with logistic regression and cross-validation.

Explore how data size and dataset complexity interact to affect learning; more data isn’t always needed or sufficient, as noise, features, and model capacity shape difficulty.