Feature Selection for Machine Learning

Learn what feature selection is and why to use it, and discover how a subset of relevant features improves interpretability, reduces training times, and enhances generalization in machine learning.

Explore filter, wrapper, and embedded feature selection methods—covering variance, correlation, lasso, and tree-based importance—and compare speed and accuracy for practical model building.

Discover filter methods for feature selection: model-agnostic, univariate ranking using statistical criteria, and fast reduction of constant, duplicated, and correlated features as the first pipeline step.

Explain wrapper methods for feature selection and contrast them with filter methods. Show forward selection, backward elimination, and exhaustive search explore subsets and discuss stopping criteria and computation costs.

Explore embedded feature selection, which integrates with model training to balance accuracy and efficiency, leveraging lasso, tree-based importances, and regression coefficients to identify key features.

Explore feature selection techniques for regression and classification using hands-on Jupyter notebooks that load, split, and select features on the training set, then evaluate on train and test sets.

Explore open source python packages for feature selection, including scikit-learn, mlxtend, and feature-engine; learn how fit and transform enable feature selection within pipelines, alongside manual pandas and numpy approaches.

Explore basic filter methods for feature selection by removing constant, quasi-constant, and duplicated features. Reduce the feature space and streamline modeling by applying these steps as sanity checks on variables.

Discover how to remove constant features—variables with zero variance—using variance threshold, pandas std, and n_unique in Python, preparing data for feature selection.

Identify and remove quasi constant features with variance threshold and value_counts, applying to numerical and categorical variables to reduce dimensionality in a notebook with pandas and scikit-learn.

Identify and remove duplicated features in a data frame using a looping approach that builds a dictionary of duplicates, prints paired features, and drops the redundant columns.

Learn to automatically drop constant and quasi-constant features with Feature-engine's DropConstantFeatures by fitting on the data and transforming to remove features with a single value.

Apply feature-engine to drop constant and quasi constant features with a 0.998 tolerance, then drop duplicates using DropConstantFeatures and DropDuplicateFeatures in a scikit-learn pipeline.

Explore correlation and its role in feature selection for machine learning. Use Pearson's correlation coefficient for linear relationships, and non-linear measures like Spearman and Kendall, via Pandas' corr.

Learn correlation feature selection, choosing predictors highly correlated with the target but uncorrelated with each other, measured by Pearson's coefficient to reduce dimensionality and preserve interpretability.

Learn two correlation-based feature selection methods: a brute force approach that removes correlated features as they appear, and a group-based method selecting one best feature from each group.

Learn Pearson correlation-based feature selection by building a Pearson matrix and visualizing with Seaborn heatmap. Use brute-force and group-based methods to prune correlated features and pick best via model importance.

Apply correlation-based feature selection to keep predictors highly correlated with the target while uncorrelated with each other, reducing feature space and improving interpretability, using Pearson's correlation coefficient.

Explore statistical filter methods for feature selection, ranking features against the target using information gain, Fisher score, univariate tests, ROC-AUC, or root mean squared error, for classification or regression.

Learn how mutual information measures the dependency between variables, use joint and marginal probabilities, and apply scikit-learn transformers like mutual_info_classif, mutual_info_regression, SelectKBest, and SelectPercentile to rank and select features.

Compute mutual information between features and the target with scikit-learn, rank features, and reduce the feature space using SelectKBest and SelectPercentile for classification and regression.

Explore the chi-square test for feature selection in machine learning, testing independence of categorical features with the target and computing the expected frequencies and p-values.

Rank categorical features in the Titanic dataset using the chi-squared test with scipy's ch2_contingency, building contingency tables via pandas cross tab, and select the top features by p-value.

Explore using the chi-square test for feature selection, including independence and expected-frequency assumptions. Learn when to use Fisher's exact test for small samples and grouping rare categories.

Explore how the expected frequencies in a chi-square test are obtained by multiplying marginals and dividing by the total observations, with joint and marginal probabilities illustrated.

The analysis of variance (anova) test evaluates whether two or more samples share the same mean, under independence, normality, and equal variance assumptions, using p-values for feature ranking in scikit-learn.

This lecture demonstrates computing p-values and f-scores with f_classif for classification and f_regression for regression, ranks features with SelectKBest and SelectPercentile, and interprets p-values and linearity assumptions using real datasets.

Rank features by p-values using scikit-learn selectors such as SelectKBest and SelectPercentile. Apply false positive rate, false discovery rate, or family-wise error controls, and consider Bonferroni or Benjamini-Hochberg corrections.

Explore feature evaluation using performance metrics in filter methods, ranking features by predictive performance. Learn two approaches: single-feature models for each feature and target-encoded representations from the KDD 2009 approach.

Train a univariate model for each feature and evaluate its performance with a chosen metric. Rank features by the metric and select the top predictors for the final model.

Explore univariate model performance using ROC-AUC and mean squared error to screen features with a decision tree, demonstrating feature selection for both classification and regression tasks.

Learn target mean encoding for feature selection in machine learning, using train-test splits to map categories and price intervals to mean targets, rank by performance, and select top features.

Learn how to perform mean encoding based feature selection for the Titanic dataset using KDD 2009, encoding categorical variables with target means, binning ages and fares, and evaluating with ROC-AUC.

Learn to perform univariate feature evaluation with the SelectBySingleFeaturePerformance transformer, using random forests to score features with ROC-AUC and R square across classification and regression, and to drop non-predictive features.

Explore target mean encoding with feature_engine by using SelectByTargetMeanPerformance to select both categorical and numerical features based on ROC-AUC, with discretization and cross-validation on the Titanic dataset.

Explore wrapper methods of feature selection as greedy search algorithms that select feature subsets to maximize a classifier's performance, including step-forward, step-backward, and exhaustive search.

Discover how the step-forward feature selection algorithm builds models with one feature, then iteratively adds features and evaluates performance to reach the optimal feature set.

Compare sequential feature selection in mlxtend and scikit-learn, showing forward and backward approaches with a Sequential Feature Selector class, and recommending scikit-learn for speed and features.

Apply step-forward feature selection with a sequential feature selector to identify the top features for a random forest, selecting 10 features using roc_auc and 3-fold cross validation.

Demonstrate step-backward feature selection on a four-feature dataset by evaluating feature subsets and selecting the best model, with user-defined stopping criteria and MLXtend Python implementation.

Demonstrates step-backward feature selection using a sequential feature selector with random forests, 3-fold cross validation, and roc_auc to select 15 features from a 57-feature dataset.

Exhaustive feature selection evaluates all feature subsets, builds models, and selects the best performer, though it is very compute-intensive. MLXtend bounds the search with a min-max feature range.

Demonstrate the exhaustive feature selection algorithm with an exhaustive feature selector for a random forest, evaluating all combinations of 1–4 features using 2-fold cross-validation on the Kaggle house price dataset.

Explore how regression coefficients indicate feature influence in linear models, and how standardization and regularization affect their reliability for feature selection.

Learn to select features using logistic regression coefficients, emphasizing proper scaling, normalization, and model assumptions, then apply select from model with get_support and transform to build a relevant feature subset.

Select important features for regression by using the magnitude of linear regression coefficients with SelectFromModel in scikit-learn, applying a mean-coefficient threshold that selects seven features in a house price dataset.

Explore how regularization alters logistic regression coefficients as penalty strength changes, using standardization and the C parameter to reveal how predictors' apparent importance shifts.

Learn how regularization reduces overfitting in linear models by penalizing coefficients; compare L1 (lasso), L2 (ridge), and elastic net, with lasso enabling feature selection by shrinking some coefficients to zero.

Explore how lasso (L1 regularization) shrinks coefficients to zero to perform embedded feature selection during model training, using scikit-learn's SelectFromModel with logistic regression and Lasso, plus comparison with L2.

Explore feature selection with tree-based methods, using random forests and decision trees to measure variable importance via impurity decrease and information gain or entropy for classification.

Demonstrate feature selection using random forest importances and SelectFromModel to train a model and identify features above the mean threshold for binary or continuous targets.

Explore recursive feature elimination with a random forest and scikit-learn, compare it to SelectFromModel, and learn stopping criteria and how to select a target number of features.