Decision Trees for Machine Learning From Scratch

Explore the ID3 overview by calculating entropy and information gain from a labeled dataset with features like outlook, humidity, and wind, and select the root node through divide-and-conquer.

Implement entropy calculation for an ID3 decision tree from scratch using pandas to load a 14-row, 5-column dataset, compute an entropy of 0.94, and define a reusable calculate_entropy function.

Learn to compute entropy and information gain to select the most dominant feature for the top node in an ID3 decision tree, then build sub datasets and continue.

Develops an id3 decision tree by implementing entropy and information gain, splitting data by outlook: sunny, rain, and overcast, and recursing to build sub data sets until decisions emerge.

Extend the ID3 decision tree to new data sets by specifying column names and loading data from csv, exemplified with the car data and generated rules.

Compute feature importance for a decision tree by evaluating information gain-based entropy reductions across features (outlook, wind, humidity) and normalize results to reveal wind and humidity as top contributors.

Explore how C4.5 extends ID3 by introducing split info and gain ratio, using entropy and information gain, base-2 logarithms, and normalization for decision tree building.

Explore how the gain ratio calculation refines decision trees by dividing information gain by split info, contrast id3 with c4.5, and show how gain ratios stabilize results.

Learn how to handle continuous features in decision trees by converting numeric values to nominal thresholds, applying gain ratio and binary splitting in C4.5, improving over ID3.

Extends the C4.5 algorithm to handle both string and numeric features, using the iris dataset to illustrate feature loading, dropping, and final decision by the most frequent class.

Transform decision rules into python-like if statements using a root variable, preconditions, and a format_rule function to print rules. Illustrate with petal length and width checks leading to iris setosa.

Learn how cart uses the gini index, calculated as 1 minus the sum of squared class probabilities, to determine the lowest gini value split for outlook, humidity, and wind.

Explore CART for classification by implementing the Gini index to identify the best split on a golf data set, handling nominal and continuous features.

Explore regression trees using standard deviation based splits. The lecture covers global, weighted, and reducted standard deviations to select the best feature and build a regression tree, with Python planned.

Explore regression trees, a CART variant for numeric targets, using standard deviation-based splits instead of entropy. Compare with ID3 and C4.5, and apply termination rules to prevent overfitting.

Explore CHAID decision trees, using the chi-square (G-squared) statistic to identify the most dominant feature for classification, building a root node and expanding with outlook and yes/no splits.

Explore how random forest mitigates overfitting by training multiple decision trees on subsampled data and using majority voting to improve unseen data classification.

learn gradient boosting machines, gradient boosting decision trees, by training trees on residual errors and summing predictions to minimize mean squared error with gradient descent and a learning rate.

Build gradient boosted models with LightGBM by converting nominal features to numerical values, preparing X and y, training the model, and inspecting feature importance and a sample decision tree.

Learn to build an XGBoost binary classifier from a dataset using label and one-hot encoding, and interpret feature importance and prediction probabilities.

Compare LightGBM and XGBoost by contrasting leaf-wise versus level-wise growth, pruning, and handling of categorical features to explain speed, robustness, and model quality.

Install the shave boost framework with pip or git clone, verify the installation by importing from shave boost, and run the dispatcher to build a decision tree and make predictions.