Data Mining for Business Analytics & Data Analysis in Python

Master survival analysis as a time-to-event approach for subscription dynamics, risk assessment, and life expectancy, with practical Python applications across insurance, banking, and healthcare.

Explore survival analysis for subscription businesses, using survival curves (customers vs time) to optimize engagement and voucher timing. Learn through a case-study approach with a step-by-step guide to get hands-on.

Study a case briefing on advanced lung cancer survival, using Kaplan-Meier curves, the log-rank test, and a four-step guide to compare male and female differences.

Mount google drive in google colab and change directory to the survival analysis folder using %cd, then organize your python notebook with sections and text.

Install and import essential Python libraries for data analysis, including lifelines for survival statistics and pandas for data manipulation, then prepare to load data for analysis.

Load data with pandas read_csv and preview it with dataset.head to explore a lung cancer survival dataset, including age, sex, weight loss, calories, status, and time to event.

In Python, transform the dependent survival variable by mapping status 1 to 0 and 2 to 1 using bracket indexing, then preview with head for analysis readiness.

Learn how the Kaplan-Meier estimator computes daily survival probabilities and builds a downward survival curve, applying events over survivors for churn and renewals.

Explore censoring in survival analysis, including right, left, and interval censoring with coronavirus and subscription examples, and preview building a Kaplan-Meier estimator in Python.

Build a Kaplan-Meier estimator with lifelines, fit the model using durations and event status, and interpret the survival curve and censoring in the dataset.

Calculate probabilities for specific events in python, explore day-by-day survival using at risk and observed counts, and approximate Kaplan-Meier survival curves.

Visualize a Kaplan–Meier survival curve with Matplotlib, add titles and axis labels (days and survival), and adjust the figure size while noting the early 5 percent survival.

Plot survival curves from a cumulative perspective in Python to visualize cumulative density and total outcomes. Learn how to compare sexes with the log rank test and extend Kaplan-Meier estimation.

Explore the log rank test to compare two groups with survival type data, test the null hypothesis of no difference, and interpret p-values against 0.05 and visualize confidence intervals.

Learn to subset a dataframe by gender in Python, creating male and female dataframes with df.query, and prepare for Kaplan-Meier survival curves.

Apply the Kaplan-Meier estimator per gender to fit male and female survival models using durations and event observed, then plot survival curves in Python.

Plot survival curves by gender, compare male and female curves, and label them to visualize differences before conducting a statistical test in the next video.

Explore survival analysis with Stephan Jenkins resources, then transform solder data to binary, fit Kaplan-Meier, plot survival curves, and perform multivariate log-rank tests in Python.

Learn to apply survival analysis in python with lifelines: transform the survival variable, fit a Kaplan–Meier estimator, plot survival curves, and perform a multivariate log-rank test across three panels.

Explore the Cox proportional hazards model as a regression tool for data mining that models time to event, includes drivers, and supports driver analysis in business like subscriptions and finance.

Explore how the Cox proportional hazard regression links baseline hazard with covariates. Interpret hazard ratios (exp coefficients) to show how covariates raise or lower risk, with Python case study.

Follow a step-by-step guide to prepare data, run Cox proportional hazards analysis, and visualize results to interpret treatment drivers in a lung cancer case study.

Prepare and clean data for Cox proportional hazard regression in Python, using lifelines in Google Colab, and extend survival analysis with a multivariate approach.

Visualize model results with a dot plot to show which factors matter, and note that coefficients and their exponentials indicate likelihood changes above or below one, with overlap signaling non-significance.

Explore extra resources and the challenge for survival analysis, including time-varying covariates and Cox proportional hazards, using the veterans lung cancer data; transform cell type to dummy variables.

Install libraries, create dummy variables with pandas get_dummies, and avoid the dummy variable trap. Fit a Cox model with lifelines, using time and status, then plot and interpret effects.

Jade, chi squared automatic interaction detection, enables driver analysis and segmentation. Direct marketing, customer segmentation, and employee and customer satisfaction studies show age as a key variable; turnover-focused practice awaits.

Explore segmented driver analysis in data mining by using a two-variable matrix to see how predictors influence a dependent variable, with Shades for group-specific relevance in Python.

Explore installing libraries for data mining in Python using Google Colab or Anaconda, with a browser-based integrated development environment, via two pip install batches including Orca and visualization tools.

Import pandas and statsmodels, prepare data, and load a dataset from statsmodels data sets for model training. Identify ethnicity as the dependent variable and discard the name to reduce noise.

Explore how CHAID builds a decision tree to rank drivers by importance and segment outcomes, using honours, computer skills, education, and resume quality to predict callback likelihood.

Explore how CHAID uses chi-squared tests on categorical variables to identify the most influential drivers, such as honors and owners, guiding data-driven segmentation.

In this Python data prep video, remove the name column, check for features with multiple unique values, and identify dummy-encoding candidates, including distinguishing numerical variables like jobs and experience.

Visualize the jobs variable with seaborn KDE plots in Python, apply a white grid style, and compare two binning approaches: simple 3–4 split and specific interval bins.

Transform the jobs variable into bins with Pandas cut, assign labels, and factorize to create a numeric categorical input, explaining why this contrasts with continuous variables.

Transform the experience variable by binning into three quantile-based groups using pandas cut with q=3, then factorize to numeric labels and explore turning job categories into dummy variables.

Transform the minimum experience variable into four bins using a Python replacement dictionary, replacing text and numeric values with standardized numeric categories for subsequent data encoding.

Transform the remaining variables into dummy variables using pandas get_dummies with drop_first to avoid the dummy variable trap, then inspect the dataset to confirm all features are numeric.

Identify the dependent variable and all independent variables using y name and x names from the dataset. Verify the 34 predictors and prepare for creating shade in the next video.

Create a CHAID model from a data frame by specifying independent and dependent variables and labeling ordinal versus nominal predictors, then set a three-level tree with 250 minimum and visualize.

Visualize and interpret a CHAID model in Python to reveal how variables like special skills, city, and email influence resume callbacks.

Explore shade, chi square automated interaction detection (chaid), and classification and regression trees through a police stops bias challenge, transforming the string variables into dummy variables and visualizing in Colab.

Create a dataset with five variables, convert strings to dummy variables, and train a depth-two decision tree to explore how race, gender, and police precinct influence vehicle search.

Explore cluster analysis, an unsupervised learning method to group observations and enable customer segmentation with the Gaussian mixture model, which assigns probabilistic cluster memberships.

Explore clustering through a country segmentation case study, determine the optimal number of clusters using AIC, visualize results, and interpret Gaussian mixture modeling outcomes for actionable insights.

Explore the Gaussian mixture model as a probabilistic clustering method that assigns soft probabilities to overlapping observations and adapts to elliptic shapes, outperforming K-means without data standardization.

Set up a Google Colab workflow by mounting drive, changing the working directory to the country-data dataset, and importing numpy, pandas, matplotlib, and sklearn gaussian mixture modeling tools.

Load data with pandas read_csv from a working directory, inspect numerical features from child mortality to GDP per capita, drop the country column, and prepare for clustering in Python.

Learn to select the optimal number of clusters with AIC and BIC, balancing simplicity and fit to avoid overfitting, using the gaussian mixture model.

Apply the Gaussian mixture model with a selected number of components (e.g., four) and a fixed random state, then fit the model to the dataset.

Explore clustering with gaussian mixture models by predicting country clusters and attaching them to the data frame, then examine cluster probabilities for each observation in Python.

Compute and display each observation's probability of belonging to every cluster using a model's predict prob, build a rounded pandas data frame, and join it with the data via pd.concat.

Explore gaussian mixture models and the infinite gaussian mixture model, using bayesian clustering to determine the optimal number of clusters for wine quality data.

Learn how to apply Gaussian mixture models in Python to discover optimal clusters, fit models on wine data, and interpret cluster profiles to assess quality attributes.

Provide feedback to turbocharge the course by filling out the feedback form in the next lecture, sharing what fires you up and what's missing the mark.

Discover the dimension reduction technique that transforms high-dimensional data into a few new predictors, addresses multicollinearity, and serves as a preprocessing step for forecasting models in business analytics.

Explore the problem of dimension reduction: reducing noise and overlapping information from multiple predictors, addressing multicollinearity and overfitting, and creating interpretable new components for models and time series forecasting.

Principal component analysis reduces data by linearly transforming variables into components, offering interpretability but potentially missing nonlinear structure; choose components using the variance explained and marginal contribution.

Learn how to import libraries in Python and apply dimension reduction with PCA in data mining for business analytics, including setting up Google Colab and mounting Google Drive.

Transform string variables into dummy variables using pandas get_dummies on the animal and furniture columns, drop the first category to avoid the dummy variable trap, and preview with head.

Build a correlation matrix in Python using Seaborn heatmap, isolate predictors, drop highly correlated columns, and interpret strong correlations to guide feature selection.

Learn to determine the optimal number of principal components by plotting explained variance and stopping when the marginal variance drops below 0.05, using four components on scaled data.

Visualize the cumulative explained variance with numpy dot and cumulative sum to decide the number of components. The fourth component explains about 95%, signaling readiness to apply PCA.

Apply principal component analysis in Python to build a four-component PCA model with scaled data, interpret components, and prepare a labeled data frame for city living characteristics.

Learn to create and name principal components in Python, interpret them for city characteristics and other features, and combine them into a data set for dimension reduction and predictive modeling.

Explore manifold learning as a non-linear dimension reduction approach that captures relationships PCA misses, focusing on t-SNE to preserve local structures in high-dimensional data, with easy implementation yet heavy computation.

Implement t-SNE from sklearn for manifold learning: create the model, set two components, apply fit_transform to X, and prepare a two-column output for visualization next video.

Visualize manifold learning outputs with t-sne scatter plots, using color-coded components and a dependent variable (rent amount) to assess dimension reduction, and explore potential improvements like random forest.

Explore online dimension reduction resources - principal component analysis, and manifold learning algorithms - and the abalone dataset challenge with data prep, dimensionality reduction, and visualization in Google Colab.

Explore data prep, dummy variables, correlation analysis, and standardization, then apply PCA and t-SNE to interpret components and visualize dimension reduction.

Utilize association rule learning to analyze customer purchasing patterns and optimize product placement, increasing baskets and improving store navigation for customers.

Open a Google Colab file, mount Google Drive, set the working directory, and import pandas and association rules from mlxtend.frequent_patterns to explore association rule learning.

Load datasets with pandas read_csv, inspect the groceries data with head, and explore association rule learning to uncover patterns of items bought together for bundling.

Explore association rule learning to uncover patterns like if a then b, measure support, confidence, and lift, and apply this interpretable recommender system to customer baskets.

Create a transaction list from the dataset by looping through rows and columns to prepare data for association rule learning in Python.

Encode transactions with a transaction encoder, fit and transform the transaction list, and convert results into a tidy pandas data frame ready for association rule learning.

Apply the apriori algorithm to boolean data frames to identify frequent item sets with a minimum support and a confidence threshold, noting its slow performance on large datasets.

Apply association rule learning with the apriori algorithm to remove the none column and analyze transactions, set minimum support of 0.03 and confidence of 0.3, then sort by lift.

Explore extra resources on association rule mining, compare algorithms, and apply association rule learning to analyze New York City restaurants by cuisine, borough, and sanitary grade, including encoding and visualization.

Learn to create transaction data, convert it into a boolean dataframe using a transaction encoder, and mine association rules with apriori in python, evaluating with minimum support, confidence, and lift.

Explore explainable AI with random forest for driver analysis to identify which drivers matter most when signing up customers through telemarketing.

Explore a telemarketing case study using random forest to identify the key drivers of customer sign-ups, assess model performance with training and test sets, and determine driver importance.

Set up a Google Colab workflow, mount Drive, and import libraries to prepare data for a random forest analysis on a bank dataset with age, job, and marital status.

Import and inspect a semicolon-delimited dataset with Pandas, fix the separator, view the head to explore age, job, marital status, housing loan, personal loan, contact details, and campaign outcome.

Transform categorical variables into numerical features using pandas get_dummies and drop first to avoid the dummy variable trap, and preview the upcoming random forest.

Explore random forest, an ensemble method that averages many decision trees to boost predictive power for classification and regression, with train-test splits to guard against overfitting.

Learn to split a dataset into training and test sets by isolating x and y, then apply train_test_split with a 0.2 test size and stratify by y to build models.

Build a random forest classifier, train it on X_train and y_train, and set up the workflow for assessing classification models in the next video.

Explore the confusion matrix for binary classification, detailing true positives and true negatives alongside false positives and false negatives, and relate accuracy, precision, recall, F1-score, and ROC AUC.

Generate predictions in Python from the trained model, compare them to actual outcomes on the test set, and address data imbalance by favoring the f-score over roc auc.

Learn to evaluate a classification model using a confusion matrix and classification report, interpret accuracy with sensitivity and F1 score, and note dataset balance when comparing models.

Demonstrate computing feature importance, plot top variables like duration and balance with horizontal bar charts, and emphasize documenting variable meanings for business analytics.

Explore extra resources on random forest applications, from remote sensing to pose recognition, and tackle a hands-on challenge transforming an extramarital affairs dataset into a binary classification with feature importance.

Isolate x and y, binarize y, and create dummy variables to prepare data for a random forest. Identify top drivers such as age, education, marriage rating, religiousness, and years married.

Explore Lime, an algorithm that explains artificial intelligence and individual predictions, using a random forest base to demonstrate its explanatory power in explainable AI.

Explore explainable ai with lime, a local interpretable, model-agnostic method that uses surrogate linear or tree models to explain individual predictions in classification tasks.

Learn to prepare a lime explainable AI workflow using a random forest on the bank telemarketing dataset in Python, including data preprocessing, model training, and lime tabular explainer.

Learn to explain model predictions with a lime explainer, visualize probabilities as a waterfall, and interpret features like balance and duration that drive yes or no outcomes.

Examine lime for model-agnostic interpretability, discuss stability improvements, and tackle a hands-on challenge predicting remote work with a random forest and lime explanations in a Google Colab workflow.

Install lime, prepare data with get_dummies, and fit a random forest in Python; then explain predictions with lime tabular explainer and isolate X and y.

Master Shap and XGBoost to explain single predictions and overall models with game theoretic Shapley values, and understand why XGBoost is fast, accurate, and competition-ready.

Build an XGBoost model to predict car prices, then use SHAP for driver analysis, following a step-by-step data preparation, training, evaluation, and interpretation process.

Learn to set up a Python environment in Google Colab, install shap, mount drive, and import key libraries like Pandas and XGBoost for data mining in business analytics.

Load the car prices data with pandas read_csv, view features like price and mileage, drop the unnamed first column, then apply xgboost and extract shap insights.

Discover XGBoost, the extreme gradient boosting algorithm, featuring a tree-based approach over linear models, a powerful ensemble with learning rate tuning for regression and classification.

The lecture visually explains how XGBoost updates weights across successive trees, increasing focus on hard-to-predict observations, while subsampling avoids overfitting and improves accuracy.

Explore how XGBoost handles regression tasks by weighting observations and predictors by error, guiding successive trees (x one, x two, and x three) to reduce error and improve accuracy.

Explore XGBoost quirks, including how feature names become information and how XGBoost captures non-linearity to powerfully analyze data in business analytics with Python.

Learn to isolate y and x during data preparation by using the first column as y and the remaining columns as x in Python.

Create a training and test split with sklearn's train_test_split, yielding X_train, X_test, y_train, and y_test at an 80/20 split with random state 1502, noting Xgboost's unique matrix requirement.

Create xgboost matrices by building a training matrix from X_train and Y_train and a test matrix built similarly. Import Xgboost as Xgb and use DMatrix to assemble the data.

Explore XGBoost parameters such as minimum child weight, learning rate (eta), max depth, gamma, subsample, column sample by tree, and number of rounds.

Create a parameter dictionary for an XGBoost regression model, detailing learning rate, max depth, column sample, sub sample, random state, evaluation metric rmse, and square error objective.

Build an XGBoost model in Python by configuring parameters, preparing dtrain and a test matrix, and setting verbosity to monitor training and the root mean square error.

Evaluate regression models by measuring prediction errors with mean absolute error and root squared mean error to guide improvements and communicate MAE insights to stakeholders.

Generate predictions with a regression model, compare actual data to forecasted values, and begin model assessment by predicting test data and measuring accuracy.

Compute mean absolute error, mean squared error, and root mean squared error with sklearn and numpy; compare mae and rmse, and introduce shap insights and xgboost.

Learn how Shap reveals the marginal contribution of each feature to model predictions, using global and local interpretability and dependency plots to analyze car price predictions.

Prepare shap by importing shap, creating an explainer object, applying three explainers to the model, and computing shap values for the X_test to enable local interpretability.

Apply local interpretability with shap values and force plot to explain car price predictions using X_test and a base value from the explainer.

Explore Python-based dependency plots to uncover relationships between variables like mileage and car value, compare continuous and binary features, and interpret cylinder effects.

Learn global interpretability with shap dot summary plots in Python, ranking features by importance and showing how high-value features positively or negatively impact price, using cylinders, mileage, doors, and convertibles.

Explore Xgboost and Shap resources and apply them to detect accidents through feature analysis, then tackle house price drivers by preparing data and building interpretability plots.

Prepare data with pandas get_dummies, split X and y, and train an XGBoost model with tuned parameters; then generate Shap explanations and visualize local and global feature effects.