Data Science:Data Mining & Natural Language Processing in R

Read csv and txt data into R and RStudio using read.csv and read.table, set the working directory, and import excel files with read.excel after installing the readxl package.

Learn to read data from SQLite databases in R by connecting with DBI and RSQLite, listing tables, reading a table, filtering results, and disconnecting.

Learn to read json data in R by using the rjson package and fromJSON, then extract id and iso 2 codes from World Bank json, with lapply and do.call rbind.

Learn to read online CSV data in R using the Oracle package, handling metadata lines, headers, and skips; extract country, salary, happiness and prepare for indexing and sub-setting.

Read data from online HTML tables, such as Wikipedia's 2016 Summer Olympics medal table, by loading the XML and Oracle libraries, retrieving the URL, and parsing with readHTMLTable.

Read html tables from wikipedia pages with the harvest library in R, applying xpath and html_table_fill to extract medal tally and UK world heritage sites data.

Use the foreign package in R to read data from other formats such as DBF and SPSS. Read DBF files tied to vector GIS shape files and explore SPSS data.

Learn to read data for data mining from formats, excel, sqlite, SPSS, and shape files, plus web scraping with rvest; start pre-processing by cleaning data and addressing missing values.

Learn to tackle missing data in R by removing NAs with na.omit or complete.cases, replacing with the mean, and performing advanced imputation with mice, including visualizing patterns.

Clean a real csv in r by removing dollar signs and commas from GDP per capita, converting to numeric, selecting columns, renaming headers, and exporting a csv.

Explore exploratory data analysis in R by visualizing distributions with histograms and box plots, and relationships with scatter plots, using iris and MP Cars data; learn ggplot2 basics.

Learn to perform exploratory data analysis in R using the XDA package, summarizing quantitative and qualitative data with num_summary and qual_summary on the Iris data, and plotting relationships by species.

Explore dplyr for data summarizing and exploration using the air quality data, learning to select and filter columns, mutate new variables, and summarize groups by month.

Learn to use dplyr for data summarizing through the pipe operator, chaining select, head, group_by, summarize, and filter on air quality to compute means by month.

Learn to use dplyr and ggplot to explore the corruption perception index with real data, creating a 2016 bar plot that shows top and bottom countries in blue and red.

Master date handling in R by parsing various date formats, converting strings to dates, and using POSIXct with time zones to support time series data.

Learn to streamline date and time preprocessing in R with the lupre date package, performing ymd, mdy, dmy conversions, make_date_time, and extracting year and month for standard formats.

Learn to plot temporal data in R by converting stock data to a time series with start and 12 monthly frequency, then visualize trends across stocks.

Visualize temporal data to detect abrupt changes in GDP growth using change point and break point detection on Japan's 1985–2015 time series, highlighting mean and variance shifts.

Examine the relationship between two quantitative variables using scatter plots and correlation, illustrating positive and negative associations while noting that correlation does not imply causation.

Learn to measure and visualize correlation in r using pearson, spearman, and kendall methods, interpret results, test significance with cor.test, and note that correlation does not imply causation.

Explore chi-square tests for independence on nominal data, build contingency tables from survey and student datasets, interpret p-values, and measure association with Phi and Cramer's V.

Use chi-squared tests to assess independence of nominal variables and measure the strength with Cramer's V, illustrated by sex and school support showing an association (Cramer's V = 0.156).

Explore data mining as the art of discovering patterns in large data sets. Learn association mining and practical use of Apriori and Eclat algorithms with support, confidence, and lift.

Explore the Apriori algorithm for association mining in R, generating frequent item set rules with support, confidence, and lift, and applying them to groceries data for purchase insights.

Apply apriori association mining to a real mushroom dataset, converting csv to sparse transactions and uncovering patterns linking gill color, cap shape, and habitat to poisonous or edible.

Visualize association rules from groceries data in R using the A rules package, exploring support, confidence, lift, and graph and matrix visualizations of high-confidence rules.

Apply the Eclat algorithm in R to the adult census data, discovering frequent item sets and associations such as capital loss none, race white, and native country United States.

Apply Eclat to the mushrooms data to mine associations among attributes, converting the data to a transactions matrix and revealing patterns like cap shape, gill attachment, and veil color.

Explore the differences between statistical modeling and machine learning, including inference and equations versus prediction on unseen data. Decide which approach fits your data based on distribution and predictive performance.

Learn the basic theory of machine learning, where algorithms learn from data without formal equations, and explore unsupervised and supervised classification, including training data and regression for continuous data.

Learn unsupervised clustering with k-means, selecting the number of clusters, initializing centers, assigning observations by Euclidean distance, and minimizing within-cluster dispersion to reveal iris data patterns.

Explore fuzzy k-means clustering in R, where data points have membership probabilities across six clusters, learn implementation with E1071, FKM, and the cluster package, and visualize and evaluate results.

Explore weighted k-means for subspace clustering on unemployment data, revealing variable weights that indicate which predictors drive cluster assignments across three clusters.

Apply the expectation maximization algorithm in r to automatically determine the number of clusters and estimate model parameters via iterative expectation and maximization steps with mclust on unemployment data.

Learn to perform unsupervised clustering with the Rattle GUI in R; read data, set seed for reproducibility, and run k-means or hierarchical clustering to visualize cluster structures.

Review unsupervised learning concepts from section six, including k-means, hierarchical, fuzzy k-means, and weight clustering, plus their visualization in R, and EM and rattle usage.

Explore dimensionality reduction and feature selection to handle multicollinearity, using ordination and principal components to produce uncorrelated predictors for regression or classification in R.

Learn principal component analysis to reduce dimensionality, generate uncorrelated components, and visualize variance explained alongside predictor contributions in data analysis.

Identify multicollinearity and remove highly correlated predictors with a 0.7 cutoff using carrot package, then validate via variance inflation factor in regression on Boston housing data.

Select influential predictors for the Boston housing data by applying lasso regression with 10-fold cross-validation, scaling and centering, and discarding zero-coefficient variables like indus and age.

Apply feature selection with the FSelector library, using correlation filter to rank numerical predictors. Explore methods for mixed data, including CFS, chi-square, and information gain, to identify important predictors.

Explore Boruta feature selection to identify predictors for malignant versus benign tumors using Boruta package and Random Forest, running 101 iterations to select 28 of 32 predictors in cancer_tumor dataset.

Conclude section seven by reviewing feature reduction and selection methods, including principal component analysis, multicollinearity handling, regression-based selection, F selector information gain, and Boruta analyses for variable relevance.

Explore basic supervised learning concepts in R, including KNN, SVM, random forests, gradient boosting, and logistic regression, with emphasis on model evaluation using AUC and cross-validation.

Prepare presence-absence data with predictors like altitude, precipitation, temperature, and land cover, and partition it into 75% training and 25% testing using caret with a fixed seed.

Explore generalized linear models for non-normal data, using exponential family distributions and link functions, with logistic and Poisson regression for binary and count outcomes in R.

Introduces logistic regression for binary response variables using a real-life voice dataset and the caret package, covering 75/25 train-test split, 10-fold cross-validation, odds ratios, and 97% accuracy.

Apply linear discriminant analysis to create a linear combination of sepal and petal measurements that separates iris species. Use LD1 and LD2 to evaluate separation and predict unseen data.

Use principal component analysis to boost binary classifiers on cancer tumor data. Compare linear discriminant analysis and naive Bayes, with PCA-enabled preprocessing.

Explore binary classification accuracy beyond overall accuracy by using confusion matrices, sensitivity, specificity, and ROC curves with AUC calculations to evaluate model performance.

Explore a multi-class loan classification problem in R, predicting loan status (paid off, collection, collection paid off) using age, education, gender, and past due dates, with NA replaced by zero.

Build and evaluate classification trees in R using caret and rpart, identifying past_due_days as the top predictor, visualizing splits, and achieving 99.2% accuracy on unseen data.

Analyze the classification tree for a binary cancer diagnosis, showing radius_worst as a split predictor and 63% benign, 37% malignant, with 93% model accuracy.

Explore how decision trees classify tumors as malignant or benign using information gain based splits. Compare J48 and C5.0 models with 75/25 training/testing in Weka, including pruning insights.

Explore how random forest models reveal individual variable influence, using a partial plot to show how past due days affects loan status outcomes such as paid off or collection.

Demonstrates building a gbm classifier for loan status with tenfold cross-validation and caret tuning, using 50 trees, depth 2, shrinkage 0.1, achieving 97% unseen accuracy; past due days is dominant.

Explore support vector machines for classification using the diamonds dataset to predict cut, compare linear, polynomial, and radial kernels, and assess performance with a 75/25 split and tenfold cross-validation.

Explore support vector machine classification with a different R package beyond caret, using ksvm with linear and RBF kernels on the diamonds data, and compare accuracy with a confusion matrix.

Conclude with a review of binary and multi-class classification, including logistic regression with/without PCA, linear discriminant analysis, trees, random forest, GBM, SVM, and model evaluation.

Use ridge regression with glmnet to model median house prices from correlated predictors, select the optimal lambda via cross-validation, and explain 96% of the variation.

Master lasso regression, using lambda to shrink and select predictors in the Boston housing data, compare with ridge, and validate with cross-validation and R-squared.

Explore generalized additive models in r to model non-linear relationships with smooth splines, compare with linear models, and interpret parametric and non-parametric terms using biocapacity data.

Master boosted generalized additive models to handle collinearity with a machine learning focus, using a 75/25 split and 10-fold cross-validation in GAMboost to predict bio capacity.

Explore multivariate adaptive regression splines (MARS), a non-parametric method that automatically models non-linearities and interactions, demonstrated on bio-capacity data with forest land, cropland, and grazing footprint as key predictors.

Explore CART regression trees in R, partition data, train with 75/25 splits, prune via cp, and visualize predictor importance for bio capacity.

Explore random forest regression, an ensemble of decision trees, and learn to assess predictor importance, perform tenfold cross-validation with caret, and visualize partial dependence plots.

Explore gradient boosting machine regression in R with caret, modeling bio capacity with predictors using data partitioning, tenfold cross-validation, and automatic parameter tuning; evaluate performance and predictor importance.

Compare regression models using a 75/25 split and 10-fold cross-validation in caret, evaluating RMSE and R-squared across gbm, mars, and rf. Mars shows strongest performance.

Explore non-parametric and non-linear regression techniques in R, including polynomial, spline, GAMs, and GBMs, and learn to evaluate, compare, and select models with train-test splits.