Hyperparameter Optimization for Machine Learning

Learn how learning algorithms optimize model parameters to minimize training criteria, while hyperparameters set outside training control capacity and improve generalization, including regularization and tuning.

Learn how to optimize hyperparameters to minimize generalization error and balance model performance with training cost. Explore hyperparameter space, sampling methods, cross-validation, and strategies like grid, random, and bayesian optimization.

Explore performance metrics essential for hyperparameter optimization, including classification and regression metrics, cross-validation, and sampling methods. Learn how scikit-learn metrics are used and how to create custom metrics for optimization.

Explore classification metrics for supervised models, including accuracy, precision, recall, f1 score, false positive rate, false negative rate, and roc-auc using a confusion matrix.

Explore regression metrics to evaluate continuous targets, including mean squared error, RMSE, MAE, and R-squared. See how these metrics measure prediction errors and explain variance to guide optimization with scikit-learn.

Create a custom false negative rate metric with the confusion matrix, use make_scorer, and optimize a random forest's hyperparameters by grid search on the breast cancer dataset.

Learn how to use scikit-learn metrics with grid search to optimize hyperparameters on the breast cancer dataset, by passing a metric string like accuracy or roc-auc.

Explore cross-validation to prevent over-fitting and assess generalization by splitting data into train, test, and folds, averaging performance across folds and comparing hyperparameter spaces with error estimates.

Explore bias, variance, and generalization error, and learn multiple cross-validation schemes—k-fold, leave-one-out, leave-p-out, repeated k-fold, and stratified k-fold—applied to model selection and hyperparameter tuning.

Demonstrates estimating a model's generalization error using multiple cross-validation schemes—k-fold, repeated k-fold, leave-one-out, leave-p-out, and stratified k-fold—on a logistic regression breast cancer model with scikit-learn.

Explore cross-validation for hyperparameter tuning in logistic regression using grid search on the cancer dataset, comparing ridge and lasso penalties across five-fold cross-validation.

Apply tailored cross-validation for grouped data, using Group K-Fold and Leave One Group Out to ensure new subjects are tested unseen. Use TimeSeriesSplit for time series to forecast future observations.

Apply group cross-validation and leave-one-group-out validation to estimate logistic regression generalization error, then perform a grid search to optimize its hyperparameters on the breast cancer dataset.

Explore basic hyperparameter search methods for machine learning, including manual, grid, and random search, and learn to balance search breadth with computational cost, hyperparameter types, and performance metrics.

Use manual search to identify promising hyperparameter regions, benchmark models, and prepare for grid search, while illustrating 5-fold cross-validation with logistic regression and random forests.

Explore how grid search exhaustively tests all combinations of specified hyperparameters using the cartesian product, highlighting limitations like the curse of dimensionality and manual value selection, while enabling parallel execution.

Apply grid search in scikit-learn to tune a gradient boosting classifier on the cancer dataset, evaluating 60 parameter combinations with 5-fold cross-validation to maximize ROC-AUC and identify the best settings.

Explore grid search over two hyperparameter spaces for a support vector classifier, comparing linear and rbf kernels. Identify best parameters by accuracy with 3-fold cross-validation, notably linear C=100.

Random search selects hyperparameter combinations at random from the hyperparameter space, using independent draws from a uniform distribution, and targets high-dimensional, continuous spaces more efficiently than grid search.

Apply random search to optimize hyperparameters for a gradient boosting classifier on the breast cancer dataset using scikit-learn and scipy.stats distributions, evaluating with cross-validation and ROC-AUC.

Use randomized search with scikit-optimize's dummy_minimize to optimize a gradient boosting classifier on the breast cancer dataset, defining real, integer, and categorical hyperparameters and evaluating via cross-validated accuracy.

Explore Hyperopt for hyperparameter optimization using random search, defining fmin-based objective functions and flexible spaces, track trials, and optimize xgboost on the breast cancer dataset for cross-validated accuracy.

Sequential search in hyperparameter optimization iteratively evaluates a few hyperparameter settings. Bayesian optimization guides where to sample next, treating the hyperparameter response surface φ(λ) as a black-box, non-differentiable function.

Explore Bayesian optimization for global optimization of costly black-box objective functions. Apply priors, posteriors, and acquisition functions, using Gaussian processes for hyperparameter tuning.

Learn the core of bayesian inference: reallocate probability from priors to posteriors using bayes' rule, updating beliefs with data across models and hyperparameters.

Explore joint and conditional probabilities with a dog breed hip dysplasia data set, computing marginal, joint, and conditional probabilities and introducing prior, posterior, and Bayes' Rule.

Explore Bayes' rule, linking prior and posterior probabilities via conditional and joint forms, with fraud detection examples, and apply these ideas to hyperparameters via bayesian optimization and gaussian processes.

Learn how sequential model-based optimization for hyperparameter optimization uses Bayes' rule to infer a posterior over a black-box objective, guided by a Gaussian process surrogate and an acquisition function.

Explore the Gaussian distribution’s bell shape, centered at the mean μ with spread σ and variance σ², including the standard normal N(0,1), and preview multivariate forms and a Gaussian process.

Generalize the univariate gaussian to multivariate distributions by modeling a vector x with mean mu and covariance matrix, capturing diagonal variances, off-diagonal covariances, and underpinning gaussian processes.

Explore how Gaussian processes model distributions over functions to estimate the hyperparameter response function, using mean, covariance, and multivariate Gaussian rules to derive priors and posteriors.

Explore how kernels measure similarity in Gaussian processes to predict hyperparameter responses, adjust smoothness, and guide Bayesian optimization with squared exponential and Matérn kernels.

Learn how acquisition functions guide Bayesian optimization for hyperparameter search by balancing exploration and exploitation, using methods like probability of improvement, expected improvement, and upper and lower confidence bounds.

Learn to implement one-dimensional Bayesian optimization with scikit-optimize to tune n_estimators for a gradient boosting classifier, using gp_minimize, and visualize convergence and Gaussian-process posteriors.

Explore bayesian optimization with scikit-optimize to tune a gradient boosting classifier by optimizing a multi-parameter space using gp_minimize, cross-validation, and convergence plots.

Explore how BayesSearchCV enables automatic hyperparameter search for a scikit-learn estimator using Bayesian optimization, selecting best parameters for a regression task and evaluating with mean squared error.

Explore scikit-optimize gp_minimize with alternative kernels like rbf and matern to optimize a gradient boosting classifier on the breast cancer dataset using a gaussian process regressor and 3-fold cross-validation.

Optimize convolutional neural network hyperparameters for MNIST using Bayesian optimization with scikit-optimize, defining the search space for learning rate, dense layers, neurons, and activations, and evaluating model performance.

Explore how scikit-optimize visualizes bayesian CNN hyperparameter searches, showing learning rate and dense nodes effects on accuracy via plots, evaluations, test set performance, and confusion matrix.

Explore sequential model-based optimization in machine learning using random forests, gradient boosting machines, and tree parzen estimators to approximate the hyperparameter response function f(x) and guide acquisition-driven sampling decisions.

Explore SMAC, a Bayesian hyperparameter optimization method that uses random forests or gradient boosted trees to approximate the objective f(x) and guide sampling via acquisition functions like expected improvement.

Optimize the convolutional neural network for MNIST digits using SMAC and Scikit-Optimize, exploring hyperparameters like convolutional and dense layers, neurons, activation, and learning rate to achieve high accuracy.

Explore Tree-structured Parzen Estimators, which model hyperparameters given the score using two densities l(x) and g(x), guiding sampling via expected improvement to focus on promising regions of the prior space.

Explore tree-structured Parzen estimators (TPE) for hyperparameter optimization by sampling configurations, splitting observations into best and rest, and modeling per-hyperparameter distributions with Parzen windows to guide expected improvement.

Explore the default hyperparameters for tree-structured Parzen estimators (TPE) in Hyperopt, including sampling, gamma quantile split, Parzen density estimation, and candidate selection.

Explore why tree-structured parzen estimators sample hyperparameters conditionally, per hyperparameter, and support nested spaces across models like svm and a decision tree.

Explore hyperparameter optimization for a convolutional neural network using Tree Parzen Estimators with Hyperopt, including learning rate, number of convolutional and dense layers, neurons, and activations.

Explore when to use basic search methods (manual, grid, random) versus Bayesian optimization (GP, SMAC, TPE) for hyperparameter tuning, including parallelization, dimensionality, and resource trade-offs.

Explore Scikit-Optimize, an open-source Python package for random and Bayesian optimization, detailing gaussian processes, forest_minimize, gbrt_minimize, and dummy_minimize, and its hyperparameter space sampling and acquisition functions.

Explore hyperparameter optimization with scikit-optimize notebooks, including randomized search and sequential methods using Gaussian processes, random forests, gradient boosting, and XGBoost, plus parallel Bayesian optimization and kernel choices.

Learn to define hyperparameter spaces with skopt.space, sampling real, integer, and categorical parameters from uniform and log-uniform distributions for machine learning models.

Define a hyperparameter space with scikit-optimize by sampling integers, real numbers, and categorical options in a param_grid for a gradient boosting machine.

Define a customizable objective function to minimize with Scikit-Optimize, passing hyperparameters to a gradient boosting classifier via named_args and set_params, using cross-validated mean performance and a negative value for accuracy.

Explore random search with scikit-optimize to tune a gradient boosting classifier, defining the hyperparameter space and objective, using dummy_minimize and 50 samples, with visualizations of convergence and evaluations.

Explore bayesian optimization with gaussian processes using scikit-optimize to tune hyperparameters, visualize with partial dependency plots, and compare against random search on the Breast Cancer dataset.

Perform Bayesian optimization with scikit-optimize using random forests as the surrogate via forest_minimize, including initial points, acquisition function, and analysis of estimators and learning rate.

Perform Bayesian optimization for machine learning using gradient boosting machines as the surrogate with gbrt_minimize. Identify how estimators, depths, learning rate, and minimum samples per split affect performance.

Learn to perform Bayesian optimization in parallel using scikit-optimize and joblib, with a Gaussian-process surrogate to sample hyperparameters and improve accuracy.

Demonstrates Bayesian optimization of scikit-learn models using BayesSearchCV to tune a gradient boosting regressor on the Boston housing dataset, comparing with grid and random search.

Explore how gp_minimize uses the Matérn kernel by default and how to switch to kernels like the RBF to better infer the Gaussian process.

Optimize cnn hyperparameters for MNIST with Bayesian optimization using Scikit-Optimize, building a convolutional neural network in Keras, preparing data, defining search space and objective, and evaluating best model and convergence.

Analyze a Bayesian optimization run with scikit-optimize's plot_objective to see how learning rate and dense layer size affect CNN accuracy, and evaluate with a confusion matrix.

Explore Hyperopt, an open-source Python package for hyperparameter optimization, offering rand.suggest, tpe.suggest, and anneal.suggest via fmin, with nested hp spaces and optional MongoDB parallelization.

Explore hyperparameter optimization with Hyperopt, covering distributions, the three sampling algorithms—random search, annealing, and tree parzen estimators—and nested hyperparameters across random forests, logistic regression, gradient boosting, and neural networks.

Learn to sample hyperparameters from hyperopt distributions, configure search spaces, and combine or nest spaces using hp.choice and pchoice.

Navigate nested hyperparameter spaces with Hyperopt, sampling naive_bayes, SVM, and decision tree algorithms and tuning C, kernel, width, depth, and minimum samples per split to find the best model.

Explore hyperparameter optimization with hyperopt, using fmin to perform randomized, annealing, and tpe searches on xgboost models, evaluating with cross-validated roc_auc.

Explore how to capture and analyze hyperparameter search progress using hyperopt, define a hyperparameter space with hp, implement objective functions, and evaluate using cross-validation, losses, and trials.

Use hyperopt to jointly optimize model choice and hyperparameters, exploring logistic regression, random forest, and gradient boosting with nested parameter spaces and cross-validated loss.

Explore Optuna, a Python package for hyperparameter optimization, covering grid and random search, nested spaces, objective functions, and analysis tools with Pandas and SQL-like storage.

Explore optuna's main functions to set up a hyperparameter search within the objective using a trial, define space with suggest_int for n_estimators and max_depth, and optimize with study.

Explore hyperparameter optimization using optuna, implementing various search algorithms, tuning scikit-learn models and neural networks, and using optuna plotting functions to analyze search characteristics across notebooks.

Explore how Optuna selects and compares hyperparameter search algorithms, including grid search, random search, TPE, and CMA-ES, and tune a random forest on breast cancer data.

Optimize hyperparameters for logistic regression, random forest, and gradient boosting classifiers with a single objective using Optuna and nested hyperparameters on the breast cancer dataset.

Optimize hyperparameters with Optuna in a Keras TensorFlow workflow on mnist, defining an objective function, sampling convolutional layers and dense layers, and evaluating accuracy to select the best model.

Explore per-layer hyperparameter sampling for a CNN with optuna and Keras sequential, varying conv and dense layer counts and tuning filters, kernel sizes, strides, and activations to maximize accuracy.

Explore Optuna's plotting capabilities for neural network hyperparameter searches, using matplotlib or plotlib backends to visualize optimization history, contour plots, edf, intermediate values, and parameter importances.