Case Studies in Data Mining with R

Name: Case Studies in Data Mining with R
Rating: 4.1 (87 reviews)

Learn to use the "Data Mining with R" (DMwR) package and R software to build and evaluate predictive data mining models.

Created byGeoffrey Hubona, Ph.D.

Last updated 8/2020

English

English [Auto],

What you'll learn

Understand how to implement and evaluate a variety of predictive data mining models in three different domains, each described as extended case studies: (1) harmful plant growth; (2) fraudulent transaction detection; and (3) stock market index changes.
Perform sophisticated data mining analyses using the "Data Mining with R" (DMwR) package and R software.
Have a greatly expanded understanding of the use of R software as a comprehensive data mining tool and platform.
Understand how to implement and evaluate supervised, semi-supervised, and unsupervised learning algorithms.

Course content

15 sections • 136 lectures • 21h 53m total length

Course Overview1:35
Advance through three extended data mining case studies: algae blooms prediction, fraudulent transactions detection, and stock market prediction, using the dmw package in R to build and evaluate predictive models.
Introduction to R for Data Mining15:57
Introduce the basics of R for data mining, including installing and loading packages, exploring data structures in RStudio, and using cheat sheets to run lm models.
Data Structures: Vectors (part 1)8:41
A vector is a sequence of data elements of the same basic type. Members in a vector are officially called components. Nevertheless, we will just call them members in this site.

Here is a vector containing three numeric values 2, 3 and 5.

> c(2, 3, 5)
[1] 2 3 5

And here is a vector of logical values.

> c(TRUE, FALSE, TRUE, FALSE, FALSE)
[1] TRUE FALSE TRUE FALSE FALSE

A vector can contain character strings.

> c("aa", "bb", "cc", "dd", "ee")
[1] "aa" "bb" "cc" "dd" "ee"
Data Structures: Vectors (part 2)9:35
Learn how to work with vectors in R: index and subsetting with brackets, assign values, and create empty and logical vectors, then vectorization and recycling.
Factors (part 1)8:20
The function factor is used to encode a vector as a factor (the terms 'category' and 'enumerated type' are also used for factors). If argument ordered is TRUE, the factor levels are assumed to be ordered. For compatibility with S there is also a function ordered.

is.factor, is.ordered, as.factor and as.ordered are the membership and coercion functions for these classes.
Factors (part 2)10:31
Generating Sequences13:42
seq() is the R function that will produce an enumerated vector.
Indexing (aka Subscripting or Subsetting)7:53
Given a vector of data one common task is to isolate particular entries or censor items that meet some criteria.
Data Structures: Matrices and Arrays (part 1)8:22
A matrix is a collection of data elements arranged in a two-dimensional rectangular layout.
Data Structures: Matrices and Arrays (part 2)7:41
An array in R can have one, two or more dimensions. It is simply a vector which is stored with additional attributes giving the dimensions (attribute "dim") and optionally names for those dimensions (attribute "dimnames").
Data Structures: Lists13:09
A list is an R structure that may contain object of any other types, including other lists. Lots of the modeling functions (like t.test() for the t test or lm() for linear models) produce lists as their return values, but you can also construct one yourself:

mylist <- list (a = 1:5, b = "Hi There", c = function(x) x * sin(x))
Data Structures: Dataframes (part 1)9:38
A data frame is a list of variables of the same number of rows with unique row names, given class "data.frame". If no variables are included, the row names determine the number of rows.
Data Structures: Dataframes (part 2)10:11
Explore data frame basics in R for mining: attach and detach, referencing columns with dollar signs, subset operations, and adding or renaming columns using built-in data sets like US arrests.
Creating New Functions12:14
Learn to create and test new functions in R, including a standard error calculator, while handling missing values with conditional logic and returning organized results.

Using the scan() Function for Input (part 1)8:20
The scan() function in R reads data into a vector or list from the console or file.
Using the scan() Function for Input (part 2)7:04
Using readline(), cat() and print() Functions12:00
The readline() function in R reads a line from the terminal (in interactive use).
Using readLines() Function and Text Data12:45
The readLines() function
Example Program: powers.r6:06
Example Program: quartiles1.r7:23
Example Program: quad2b.r8:23
Create an interactive program that uses read line to input a, b, and c for a quadratic, computes the discriminant, and prints the roots: two, one, or none.
Reading and Writing Files (part 1)5:48
Reading and Writing Files (part 2)13:42
Learn to read files into matrices with a read matrix function, converting between data frames and matrices. Handle headers and write data to disk with write, write.table, cat, and dump.

Predicting Algae Blooms12:29
This case study introduces you to some basic tasks of data mining: data pre-processing, exploratory data analysis, and predictive model construction. This initial case study studies a relatively small problem by data mining standards. Namely, the case addresses the problem of predicting the frequency occurrence of several harmful algae in water samples.
Data Visualization and Summarization: Histograms14:34
A histogram is a graphical representation of the distribution of numerical data. It is an estimate of the probability distribution of a continuous variable (quantitative variable) and was first introduced by Karl Pearson.
Data Visualization: Boxplot and Identity Plot13:15
The box plot (a.k.a. box and whisker diagram) is a standardized way of displaying the distribution of data based on the five number summary: minimum, first quartile, median, third quartile, and maximum.
Data Visualization: Conditioning Plots14:48
Conditioning Plot. Purpose: Check pairwise relationship between two variables conditional on a third variable. A conditional plot, also known as a coplot or subset plot, is a plot of two variables contional on the value of a third variable (called the conditioning variable).
Imputation: Dealing with Unknown or Missing Values16:03
In statistics, imputation is the process of replacing missing data with substituted values. When substituting for a data point, it is known as "unit imputation"; when substituting for a component of a data point, it is known as "item imputation". Because missing data can create problems for analyzing data, imputation is seen as a way to avoid pitfalls involved with listwise deletion of cases that have missing values. That is to say, when one or more values are missing for a case, most statistical packages default to discarding any case that has a missing value, which may introduce bias or affect the representativeness of the results. Imputation preserves all cases by replacing missing data with a probable value based on other available information. Once all missing values have been imputed, the data set can then be analysed using standard techniques for complete data
Imputation: Removing Rows with Missing Values11:09
Examine the apply family of functions in R to detect and handle missing values by row, using is.na and many to remove or impute rows with excessive NAs.
Imputation: Replace Missing Values with Central Measures10:04
Imputation: Replace Missing Values through Correlation13:57
Identify highly correlated variables with a correlation matrix to impute missing values. Build and apply a linear regression model to predict missing entries, excluding samples with 20% or more missing.
Visualizing other Imputations with Lattice Plots13:33
The lattice package, written by Deepayan Sarkar, attempts to improve on base R graphics by providing better defaults and the ability to easily display multivariate relationships. In particular, the package supports the creation of trellis graphs - graphs that display a variable or the relationship between variables, conditioned on one or more other variables.

Read in Data Files14:18
Creating Prediction Models15:21
Explore building prediction models for algae frequency across rivers using multiple linear regression and regression trees, handling missing values with imputation, and interpreting model outputs.
Examine Alternative Regression Models17:30
In statistics, regression analysis is a statistical process for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables (or 'predictors').
Regression Trees15:42
Regression trees are for dependent variables that take continuous or. ordered discrete values, with prediction error typically measured by the squared. difference between the observed and predicted values.
Strategy for Pruning Trees11:47
Grow a bushy tree with rpart in data mining with R, then prune via cost-complexity criteria and cross-validation to balance predictive accuracy and tree size.

Alternative Model Evaluation Criteria14:14
Compare regression and regression tree models using mean absolute error, mean squared error, and normalized mean squared error to assess predictive performance on algae growth.
Introduction to K-Fold Cross-Validation11:32
Cross-validation, sometimes called rotation estimation, is a model validation technique for assessing how the results of a statistical analysis will generalize to an independent data set. It is mainly used in settings where the goal is prediction, and one wants to estimate how accurately a predictive model will perform in practice. In a prediction problem, a model is usually given a dataset of known data on which training is run (training dataset), and a dataset of unknown data (or first seen data) against which the model is tested (testing dataset). The goal of cross validation is to define a dataset to "test" the model in the training phase (i.e., the validation dataset), in order to limit problems like overfitting, give an insight on how the model will generalize to an independent dataset (i.e., an unknown dataset, for instance from a real problem), etc.
Setting up K-Fold Evaluation (part 1)10:30
In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times (the folds), with each of the k subsamples used exactly once as the validation data. The k results from the folds can then be averaged (or otherwise combined) to produce a single estimation. The advantage of this method over repeated random sub-sampling (see below) is that all observations are used for both training and validation, and each observation is used for validation exactly once. 10-fold cross-validation is commonly used
Setting up K-Fold Evaluation (part 2)10:26
Learn how to set up and interpret a 10-fold cross-validation across three tree model variants and a linear model, using normalized mean squared error and box plots to compare performance.
Best Model (part 1)10:10
Explore data mining with R by applying vectorized supply functions to compare seven target variable datasets, using user defined functions and cross validation for robust model evaluation.
Best Model (part 2)9:52
Finish Evaluating Models11:09
Use cross-validation and argo’s core functions to compare three modeling variants with one evaluation metric, highlighting random forest performance and nonparametric Wilcoxon analysis.
Predicting from the Models10:16
The lecture demonstrates predicting seven algae outcomes using random forests and regression trees, handling missing values with imputation, and averaging predictions on the 140 test samples.
Comparing the Predictions10:16

Exercise Solution from Evaluating and Selecting Models3:59
Fraudulent Case Study Introduction3:04
This case study addresses an instantiation of the general problem of detecting unusual observations of a phenomena, that is, finding rare and quite different observations. The driving application has to do with transactions of a set of products that are reported by the salespeople of some company. The goal is to find "strange" transaction reports that may indicate fraud attempts by some of the salespeople.
Prelude to Exploring the Data4:51
Exploring the Data11:21
Data visualization is the presentation of data in a pictorial or graphical format. For centuries, people have depended on visual representations such as charts and maps to understand information more easily and quickly.
Exploring the Data Continued (part 1)13:36
Analyze unit price to reveal fraud signals across sales transactions and manage missing data. Learn to compute unit price per transaction and handle rare products in large data with R.
Exploring the Data Continued (part 2)13:24
Exploring the Data Continued (part 3)13:15
Explore how top 100 salesmen contribute nearly 40 percent of revenue and top 100 products account for about 75 percent of quantities, revealing skewed distributions and actionable insights.
Dealing with Missing Data (part 1)10:13
Identify 888 records with both value and quantity missing, recreate consistent tables, evaluate deleting those cases, and explore one-missing scenarios using product-level and salesperson analyses.
Dealing with Missing Data (part 2)7:23
Identify and remove records with missing quantity for certain products, update product levels, and assess missingness by salesperson; then impute remaining gaps using data from other transactions.
Dealing with Missing Data (part 3)10:46

Review the Data and the Focus of the Fraudulent Transactions Case12:50
Explore hands-on data mining with R by building conditional and random forest trees on the iris and fraud datasets, interpreting plots and confusion matrices, and applying a 70/30 train–test split.
Pre-Processing the Data (part 1)10:27
Pre-Processing the Data (part 2)10:39
Here we explain the whys and hows of creating a list structure containing the unit prices by product.
Pre-Processing the Data (part 3)12:28
Explore pre-processing steps in data mining with r: center and scale price distributions, run a K-S test to find similar products, and build a similarity matrix for fraud detection.
Defining Data Mining Tasks11:55
In supervised learning the categories, data is assigned to are known before computation. So they are being used in order to 'learn' the parameters that are really significant for those Clusters. In unsupervised learning Datasets are assigned to segments, without the clusters being known.
Semi-Supervised Techniques6:51
Semi-supervised learning is a class of supervised learning tasks and techniques that also make use of unlabeled data for training - typically a small amount of labeled data with a large amount of unlabeled data.
Precision and Recall7:31
In pattern recognition and information retrieval with binary classification, precision (also called positive predictive value) is the fraction of retrieved instances that are relevant, while recall (also known as sensitivity) is the fraction of relevant instances that are retrieved.
Lift Charts and Precision Recall Curves12:11
Lift is a measure of the effectiveness of a predictive model calculated as the ratio between the results obtained with and without the predictive model. Cumulative gains and lift charts are visual aids for measuring model performance. Both charts consist of a lift curve and a baseline.

Exercise from Previous Session1:16
Review Precision and Recall10:05
Review Lift Charts and Precision Recall Curves8:33
Cumulative Recall Chart10:08
Analyze the cumulative recall chart to balance recall and resource effort, identifying the sweet spot. Use normalized distance to the typical price to detect outliers and fraud.
Creating More Functions for the Experimental Methodology7:04
Explore creating additional custom functions for the experimental methodology in data mining with R, compute price distributions, normalized distance to the typical price, and outlier statistics for transactions.
Experimental Methodology to find Outliers (part 1)10:36
Explore experimental methodology for detecting outliers in data mining with R, focusing on ranking fraudulent transactions, stratified sampling and bagging, and evaluating precision and recall.
Experimental Methodology to find Outliers (part 2)11:27
An outlier is an observation that lies outside the overall pattern of a distribution (Moore and McCabe 1999). Usually, the presence of an outlier indicates some sort of problem. This can be a case which does not fit the model under study, or an error in measurement. Outliers are often easy to spot in histograms.
Experimental Methodology to find Outliers (part 3)9:12
Experimental Methodology to find Outliers (part 4)10:18
Explore an experimental methodology in data mining with r for identifying outliers, using PR curve, cumulative recall, and a nearest neighbors approach.
Experimental Methodology to find Outliers (part 5)6:57
Compare the nearest-neighbors local outlier approach with k=7 to BP rule and box plot rules, showing higher average recall with fewer resources and better precision for low recall values.

Review of Fraud Case (part 1)10:50
Identify fraudulent sales reports through data mining with R and the dmwr package, emphasizing precision, recall, and probabilistic ranking of unknown transactions. Save and load workspaces to streamline long-case analyses.
Review of Fraud Case (part 2)11:08
Review of Fraud Case (part 3)10:33
Baseline Boxplot Rule7:47
This lecture presents the baseline boxplot rule for probabilistically ranking frauds, using a 10 percent holdout with a fixed seed to compare predictions against actuals and evaluate recall.
Local Outlier Factors11:38
state-of-the-art outlier ranking method. The main idea of this system is to
try to obtain an outlyingness score for each case by estimating its degree of

isolation with respect to its local neighborhood. The method is based on the

notion of the local density of the observations. Cases in regions with very low

density are considered outliers. The estimates of the density are obtained using

the distances between cases.
Plotting Everything8:22
Compare precision-recall and box-plot visuals, showing local outlier performance yielding higher recall than the box rule. Introduce hierarchical clustering to rank outliers by perturbing homogeneity in unsupervised learning with R.
Supervised and Unsupervised Approaches11:19
From a theoretical point of view, supervised and unsupervised learning differ only in the causal structure of the model. In supervised learning, the model defines the effect one set of observations, called inputs, has on another set of observations, called outputs. In other words, the inputs are assumed to be at the beginning and outputs at the end of the causal chain. The models can include mediating variables between the inputs and outputs. In unsupervised learning, all the observations are assumed to be caused by latent variables, that is, the observations are assumed to be at the end of the causal chain. In practice, models for supervised learning often leave the probability for inputs undefined. This model is not needed as long as the inputs are available, but if some of the input values are missing, it is not possible to infer anything about the outputs. If the inputs are also modelled, then missing inputs cause no problem since they can be considered latent variables as in unsupervised learning.
SMOTE and Naive Bayes (part 1)10:49
SMOTE and Naive Bayes (part 2)10:44

Introduction to Boosting (from Rattle course)8:34
Explore boosting as a meta ensemble method that uses weak learners and misclassified observation weights to build an additive, weighted model, often via decision trees, with a practical Rattle tutorial.
Boosting Demo Basics using R9:01
Boosting is a machine learning ensemble meta-algorithm for reducing bias primarily and also variance in supervised learning, and a family of machine learning algorithms which convert weak learners to strong ones.
Replicating Adaboost using Rpart (Recursive Partitioning) Package11:10
Recursive partitioning is a statistical method for multivariable analysis. Recursive partitioning creates a decision tree that strives to correctly classify members of the population by splitting it into sub-populations based on several dichotomous independent variables.
Replicating Adaboost using Rpart (part 2)10:48
AdaBoost, short for "Adaptive Boosting", is a machine learning meta-algorithm formulated by Yoav Freund and Robert Schapire who won the prestigious "Gödel Prize" in 2003 for their work.
Boosting Extensions and Variants14:06
Explore boosting extensions and variants in data mining with R, including gbm and alternating decision trees. Build and evaluate adt models and compare performance using confusion matrices.
Boosting Exercise6:24
Apply boosting to predict wind types by iteratively upweighting misclassified observations with decision stumps, then build three models and create an ensemble from them.

Requirements

Students will need to install no-cost R software and the no-cost RStudio IDE (instructions are provided).

Description

Case Studies in Data Mining was originally taught as three separate online data mining courses. We examine three case studies which together present a broad-based tour of the basic and extended tasks of data mining in three different domains: (1) predicting algae blooms; (2) detecting fraudulent sales transactions; and (3) predicting stock market returns. The cumulative "hands-on" 3-course fifteen sessions showcase the use of Luis Torgo's amazingly useful "Data Mining with R" (DMwR) package and R software. Everything that you see on-screen is included with the course: all of the R scripts; all of the data files and R objects used and/or referenced; as well as all of the R packages' documentation. You can be new to R software and/or to data mining and be successful in completing the course. The first case study, Predicting Algae Blooms, provides instruction regarding the many useful, unique data mining functions contained in the R software 'DMwR' package. For the algae blooms prediction case, we specifically look at the tasks of data pre-processing, exploratory data analysis, and predictive model construction. For individuals completely new to R, the first two sessions of the algae blooms case (almost 4 hours of video and materials) provide an accelerated introduction to the use of R and RStudio and to basic techniques for inputting and outputting data and text. Detecting Fraudulent Transactions is the second extended data mining case study that showcases the DMwR (Data Mining with R) package. The case is specific but may be generalized to a common business problem: How does one sift through mountains of data (401,124 records, in this case) and identify suspicious data entries, or "outliers"? The case problem is very unstructured, and walks through a wide variety of approaches and techniques in the attempt to discriminate the "normal", or "ok" transactions, from the abnormal, suspicious, or "fraudulent" transactions. This case presents a large number of alternative modeling approaches, some of which are appropriate for supervised, some for unsupervised, and some for semi-supervised data scenarios. The third extended case, Predicting Stock Market Returns is a data mining case study addressing the domain of automatic stock trading systems. These four sessions address the tasks of building an automated stock trading system based on prediction models that utilize daily stock quote data. The goal is to predict future returns for the S&P 500 market index. The resulting predictions are used together with a trading strategy to make decisions about generating market buy and sell orders. The case examines prediction problems that stem from the time ordering among data observations, that is, from the use of time series data. It also exemplifies the difficulties involved in translating model predictions into decisions and actions in the context of 'real-world' business applications.

Who this course is for:

The course is appropriate for anyone seeking to expand their knowledge and analytical skills related to conducting predictive data mining analyses.
The course is appropriate for undergraduate students seeking to acquire additional in-demand job skill sets for business analytics.
The course is appropriate for graduate students seeking to acquire additional data analysis skills.
Knowledge of R software is not required to successfully complete this course.
The course is appropriate for practicing business analytics professionals seeking to acquire additional job skill sets.

Case Studies in Data Mining with R

What you'll learn

Explore related topics

Course content

A Brief Introduction to R and RStudio using Scripts14 lectures • 2hr 17min

Inputting and Outputting Data and Text9 lectures • 1hr 22min

Introduction to Predicting Algae Blooms9 lectures • 2hr

Obtaining Prediction Models5 lectures • 1hr 15min

Evaluating and Selecting Models9 lectures • 1hr 38min

Examine the Data in the Fraudulent Transactions Case Study10 lectures • 1hr 32min

Pre-Processing the Data to Apply Methodology8 lectures • 1hr 25min

Methodology to Find Outliers (Fraudulent Transactions)10 lectures • 1hr 26min

The Data Mining Tasks to Find the Fraudulent Transactions9 lectures • 1hr 33min

Sidebar on Boosting6 lectures • 1hr

Requirements

Description

Who this course is for: