Ultimate ML Bootcamp #6: Advanced Decision Tree Techniques

Combine predictions from many decision trees to form a random forest, using bagging and random subspace to improve generalizability and reduce overfitting.

Apply and optimize a random forest using a systematic grid search with five-fold cross-validation, compare accuracy, F1, and AUC, and finalize a robust model with feature importances and learning curves.

Discover gradient boosting machines and gradient descent applied to tree methods, tracing AdaBoost foundations, optimization with residuals, and additive models that form a strong, sequential predictive model.

Explore additive modeling in gbm by starting with a base predictor and iteratively modeling residuals with delta models, boosting predictions through gradient descent and residual-based corrections.

Explore hyperparameter optimization of GBM using the gradient boosting classifier, tune learning rate, max depth, n estimators, and subsample to improve accuracy, F1, and AUC.

Explore XGBoost, an optimized, scalable version of GBM that boosts gradient descent for fast predictions, with languages such as R, Python, Hadoop, Scala, and Julia and success in Kaggle competitions.

Explore XGBoost core concepts by building a model, performing cross validation, tuning hyperparameters such as n_estimators, learning rate, max depth, and subsample by tree, then assess accuracy and f1 score.

Explore Lightgbm, a gbm variant optimized for training speed using leaf-wise growth and depth-first splitting, contrasting with XGBoost’s level-wise approach to handle large data efficiently.

In this Lightgbm hands-on session, we build a model, compare speed with XGBoost, and tune hyperparameters like learning rate, n_estimators, and call_sample_by_tree, using cross-validation scores to select best settings.

Discover CatBoost, a boosting method that natively handles categorical variables, avoiding extensive one hot or label encoding, and evaluate its speed, scalability, and GPU support in practice.

Set verbose to false in CatBoost, run a lean grid search to tune iterations, depth, and learning rate, and compare final AUC with Lightgbm.

Explore feature importance across models like random forest, GBM, XGBoost, and LightGBM, highlighting top predictors such as glucose, BMI, and age, and discuss feature engineering to keep models simple.

Explore random search cross-validation for hyperparameter optimization in a random forest, comparing it to grid search CV and showing how to select 100 random candidates across a broad parameter range.

Examine learning curves to assess model complexity and overfitting with AUC, comparing training, validation, and test scores across max depth, max features, and estimators, via hyperparameter optimization.