Data Science and Machine Learning in Python: Linear models

Explore derivative functions and the slope of the original function to identify minima and maxima, using x^2 as an example and applying the second derivative test.

Explore simple linear regression to predict salary from years of experience, modeling the mean salary as a function of experience with intercept beta0 and slope beta1, assuming fixed standard deviation.

Explore the four core assumptions of simple linear regression: linearity, constant variance, independence, and normality, and how violations affect model performance.

Learn how to fit a linear regression model to salary based on years of experience by maximizing the likelihood, using beta zero, beta one, and sigma, via log-likelihood.

Explore how to estimate a linear normal model using maximum likelihood estimation, deriving beta zero, beta one, and sigma by minimizing the negative log likelihood.

Use the maximum likelihood principle to estimate beta zero by differentiating the negative log-likelihood and solving, yielding beta zero hat equals salary mean minus beta one times experience mean.

Compute beta one hat as the sample covariance between years of experience and salary divided by sample variance of years of experience, using the negative log-likelihood to isolate beta one.

Apply the maximum likelihood procedure to estimate sigma squared, derive its derivative via the log-likelihood, and show the best sigma squared equals the mean square error for salary predictions.

Interpret coefficients via maximum likelihood to build fit; beta zero is salary intercept at zero years of experience, beta one the slope computed as cov(x,y)/var(x), with standard deviation describing dispersion.

Evaluate linear model goodness of fit using mean squared error and its root, alongside the coefficient of determination, by comparing total and residual sums of squares for salary data.

Explore ordinary least squares in linear regression by defining residuals and minimizing the residual sum of squares, linking it to negative log likelihood for estimating beta0, beta1, and sigma.

Explore the impact of omitting the intercept in a linear regression, forcing the line through the origin and altering beta1 estimation, residuals, and model fit.

Evaluate model performance by analyzing residuals for normality with a histogram, Q Q plot, and scatterplots; verify linearity and constant variance across years of experience for a reliable fit.

Import numpy, seaborn, matplotlib, and a machine learning library to build a linear regression model with years of experience predicting salary. Visualize results with scatterplot and fix a random seed.

Derive the regression parameters beta zero hat and beta one hat from data using n, x mean, y mean, var(x), and cov(x,y), then estimate sigma hat.

Evaluate model goodness of fit with R2 score, MSE, and RMSE to interpret how well the salary Y variable is predicted, including RMSE of $5,711.

Compute the mean and standard deviation of the residuals, plot the normal distribution with these stats, and compare to a histogram to verify residual normality.

Evaluate model assumptions by visualizing residuals versus years of experience and salary versus predicted salary, and confirm normality with a Q Q plot, supporting a linear model for salary.

We compare estimated parameters with data-generating values in a simple linear regression, showing beta zero 39498, beta one 2566, and sigma 5711, as the model minimizes prediction errors.

Extend a single-variable model to multiple features using a design matrix and beta vector to compute salary predictions via dot products, incorporating the intercept and feature coefficients.

Find the optimal beta for a multiple linear regression by minimizing the residual sum of squares with vector notation and the ordinary least squares solution using the design matrix.

Begin implementing multiple linear regression models with two explanatory variables, visualize data in 3D scatterplots, assess model diagnostics with the variance inflation factor, and ensure reproducibility with a seed.

Learn to build a first multiple linear regression model with an optional intercept, add a bias column to the x matrix, and compute beta and make predictions.

Complete the linear regression class to build the first multiple linear regression model, add a bias column, predict Y from X with beta, and assess with R squared and MSE.

Visualize the first multiple linear regression model by plotting the plane of best fit in three-dimensional space, computing y from beta0, beta1 x1, and beta2 x2 using a grid.

Learn to include categorical variables in a salary regression by using dummy variables, a reference category, and two- or three-category coding with beta coefficients, ensuring only one dummy is active.

Learn how a dummy variable for a postgraduate degree shifts the intercept in a linear regression. See how two expected salary lines remain parallel and differ by beta_k.

Explore how feature interactions in regression use the product of variables to shift intercepts and slopes, demonstrated with postgraduate degree and years of experience affecting salary.

Extend the second multiple linear regression model with a numerical x1 and a dummy variable, and visualize data via a 2d scatterplot.

Build and fit a multiple linear regression model with a dummy variable, interpret the intercept and coefficients, and plot parallel regression lines for zero and one categories.

Standardize predictor scales using z-score transformation to make beta coefficients comparable, enabling ranking of variable importance in linear models.

Rank the predictive power of four numeric features in a linear regression by standardizing the design matrix and comparing standardized beta coefficients.

Diagnose multicollinearity in multiple linear regression by examining the design matrix and predictor relations; distinguish perfect and imperfect cases, and apply remedies like removing redundant variables, dummy variables, or regularization.

Assess multicollinearity by computing VIF for each predictor via regressing it on the others, using VIF = 1/(1-R^2), and flag values above five.

Explore how imperfect multicollinearity among x1, x2, and x3 destabilizes beta coefficients in linear regression; the lecture demonstrates ten-subset resampling and box plots to show coefficient instability.

Compute variance inflation factors (VIF) for each predictor with statsmodels, add a constant to the design matrix, and remove the x three variable to resolve multicollinearity.

Explore how treating multicollinearity by excluding a variable stabilizes linear model coefficients across ten data subsets, improving intercept and slope consistency, with visual box-plot evidence.

Define the problem, collect and clean data, perform exploratory analysis, select or engineer features, build and evaluate models, and communicate results within the data science life cycle.

Model diamond prices using a dataset with carat, cut, color, clarity, and dimensions. Load and describe the data with Pandas and plotting tools, identify outliers, and prepare for error checking.

Filter out zero x, y, and z values to correct data quality issues, drop invalid rows, and verify dataset completeness before a train-validation-test split.

Use training, validation, and test sets to build a model from the design matrix, assess generalization to unseen data, and confirm final performance on the test set before deployment.

Split the dataset into 20% test and 80% training with a train test split function, then allocate 20% of training to validation.

Visualize numerical and categorical diamond features to identify price predictors, creating scatterplots of variables like carat, depth, and table, and temporarily remove extreme outliers to reveal relationships.

Remove outliers by using the 1st and 99th percentile to define the central 98%. Price shows non-linear relationships with carats and with X, Y, Z, hinting at future transformations.

Transform predictors or the target to fix non-linear relationships, using log transforms as needed, while keeping the model linear in beta parameters.

Transform the diamond price and explanatory variables with logarithms to improve linearity. Create log price and log X, log Y, log Z to reveal clearer relationships.

Create a train data copy, recompute logarithmic variables for price and X, Y, Z, and drop non-logarithmic columns; use a heatmap to spot multiple linearity problems and keep only log(carat).

Move beyond correlation to measure dependency with mutual information, using mutual_info_regression to select strong predictors for the target, including non-linear and categorical variables, such as the logarithm of price.

Encode categorical variables as numerical labels and use mutual information to assess how cut, color, and clarity predict diamond price, then compare carat plus clarity, color, and cut models.

Learn to build a two-step pipeline with column transformer and transform target regression in Psychic Learn, performing variable selection, transformation, and model fitting via feed and predict.

Use the column transformer to select and transform data for a model, applying one hot encoding to clarity and a log transform to current, within a pipeline.

Use the TransformedTargetRegressor in a pipeline to model the logarithm of price and predict the actual price with the inverse function, via fit and predict on the design matrix.

Build three price models from design matrix with a log-transformed target; preprocess via a column transformer and one-hot encode clarity, then evaluate using R squared and root mean squared error.

Build and compare three linear models for diamond price, including log carat, color, and cat as dummies, evaluate with R-squared and RMSE, and select model two for performance and simplicity.

Explain interpreting a coefficient with log-transformed X: a one-unit increase in log X raises Y by beta1, and percent change in X maps to delta Y = beta1 * log(1+P/100).

Interpret the logarithmic model by showing that a unit x increase yields a relative change in y equal to beta1, with dy/dx = beta1*y and beta1 = (dy/dx)/y.

Learn how to interpret model parameters, especially beta one, when both the target variable and the predictors are log-transformed, via the ratio of relative changes in Y and X.

Interpret the beta coefficient for a dummy predictor in a log-transformed target model; the relative change is e^{beta1}-1, and beta1=0.2 implies about a 22.14% increase.

Evaluate the selected linear model on the testing dataset to confirm stability and similar prediction error, then extract beta parameters and interpret elasticity for log carat and color/clarity categories.

Avoid overfitting by balancing model complexity to capture underlying patterns, not noise. Use training, validation, and test sets to select and verify models that generalize to new data.

Polynomial regression extends linear models by adding polynomial features such as x squared and x cubed to the design matrix to model curvature.

Extend linear regression to polynomial regression to capture non-linear relationships, using numpy, visualization libraries, and scikit-learn to generate data, visualize X vs Y, and prepare for polynomial modeling.

Transform the design matrix with degree two polynomial features, including x squared, and fit a linear regression to capture curvature in the relationship between x and y.

Show how interaction terms in polynomial regression capture curvature by including products of variables with betas, and interpret their effect on y via derivatives in x1 and x2.

Explore polynomial regression with interaction terms between two features, visualize a three dimensional scatterplot, and build a model that captures curvature using interaction features.

Explore how polynomial regression models risk overfitting as complexity increases with exponents and interactions, and use training, validation, and test splits to select the appropriate degree.

Explore structural multicollinearity in polynomial regression when x, x^2, and x^3 are tightly related. Center data before powering x to reduce linearity and retrieve original parameters from the centered model.

Demonstrate polynomial regression and the risks of overfitting and underfitting across degrees 1, 2, 3, and 10, with validation and regularization to improve generalization.

Explore polynomial regression and structural multicore linearity in a design matrix. Center the variables to remove multicollinearity and prepare for linear regression with centered x and x squared.

Develop a linear regression model with centered polynomial features, recover intercept, beta1, and beta2 from the centered parameters, and interpret y as intercept plus beta1 x plus beta2 x squared.

Explore regularized models to curb overfitting, stabilize linear regressions, and mitigate outlier influence, contrasting ordinary least squares with regularization techniques for robust beta estimates.

Learn ridge regression, a regularization technique that adds an L2 penalty with lambda to the mean squared error to handle overfitting and multiple linearity, solved with robust singular value decomposition.

Explore ridge regression beta estimation via eigen decomposition and the economy version of single value decomposition using U, Sigma, and V^T, to avoid inverting near-zero eigenvalues and overflow with regularization.

Explore how the lambda parameter shapes ridge regression estimates through the diagonal penalty, highlighting the role of singular values and the pseudo inverse, and why standardizing the design matrix matters.

Compute the ridge intercept by minimizing the penalized mean squared error and solving for beta zero as mean(y) minus beta dot the mean feature vector (excluding the intercept).

Explore ridge regression to build robust linear models that resist multicollinearity and overfitting. Use Python visualizations to show how identical features create unstable coefficients but still yield a regression plane.

Implement ridge regression code 2 by building a Rich class with lambda, fit intercept, and tol; center data, perform SVD on X, filter singular values, compute coefficients, and generate predictions.

Video demonstrates ridge regression by replacing linear regression with a lambda-penalized model, using a slider to explore coefficient stability and how increasing lambda reduces coefficient magnitudes to handle multiple linearity.

See how outliers distort linear models and how ridge regression uses a lambda penalty to reduce slope influence, improving fit while acknowledging its limits with extreme negative observations.

Explore ridge regression to reduce polynomial regression complexity by penalizing parameter size, showing how varying lambda decreases curvature and overfitting while preserving the data trend.

Compare ordinary linear regression to ridge regression with a parameter penalty, visualizing the mean square error across beta1 and beta2 on a 3D plot.

Explore lasso regression, an L1-regularized linear model for minimizing sum of squared residuals with a lambda-weighted L1 penalty. The L1 norm is non-differentiable at zero, so no analytical solution exists.

Coordinate descent optimizes lasso regression by iteratively updating beta parameters to minimize the cost function. Starting from a random guess, it alternates updates for each beta with a step size.

Explore how coordinate descent optimizes lasso regression by updating each beta parameter in the linear regression objective, handling nondifferentiability, and deriving update formulas from residuals and X.

Apply sub derivative and sub differential concepts to lasso regression to compute the coordinate descent update for beta, handling the absolute value penalty at zero.

Explore lasso regression through subdifferentials and coordinate descent, applying soft thresholding to update beta parameters and perform shrinkage and selection with the lambda parameter.

Compare lasso and ridge regression penalties in minimizing squared residuals, where lasso uses the L1 norm and ridge uses the L2 norm. Lead to sparse or dense solutions.

Explore k-fold cross-validation to estimate a model's predictive capacity with limited data by rotating training and validation subsets, averaging results, and validating the winner on the test set.

Explore lasso regression by penalizing the absolute values of parameters to select useful variables, and learn coordinate descent to iteratively minimize the cost function from a random initial beta.

Use coordinate descent to minimize a differentiable function by alternately updating x1 and x2 with a step size alpha over ten iterations, illustrating how lasso regression finds parameters.

Define a lasso class to estimate beta parameters with lambda regularization via coordinate descent, including intercept, tolerance, and max iterations, plus fit and predict using soft-thresholding.

Observe how lambda drives coefficient paths in lasso regression using a log-spaced 20-value grid from 1e-5 to 1e3, with large lambda zeroing coefficients and smaller ones revealing predictive features.

Explore how to use lasso regression to select important polynomial features and find the optimal lambda via cross-validated mean squared error, achieving a degree three polynomial model.

Train a degree ten polynomial with lasso, vary lambda, and observe how the model fit changes on training and test data to combat overfitting, highlighting coefficient sparsity and variable elimination.

Use lasso as a preprocessing step to identify useful features in a design matrix and build a linear regression model, then compare coefficients with true coefficients.

Compare ridge and lasso regression using cost surfaces, showing ridge minimizes squared residuals while lasso yields sparse coefficients by minimizing the sum of absolute values.

Explore linear methods for classification, including binary and multiclass tasks, and compare label representations—numerical, text, and one-hot encoding—to define decision boundaries.

Explore binary logistic regression, learning to compute a class probability via sigmoid of log odds using beta parameters and features, and apply a 50% decision boundary to two-class classification.

Explore how to estimate optimal beta parameters for logistic regression by maximizing the likelihood (via negative log-likelihood) and using gradient descent to minimize the loss for Bernoulli outcomes.

This lecture explains gradient descent for logistic regression, showing how to initialize x, select a step size alpha, and update x opposite the gradient to minimize the negative log likelihood.

Compute the gradient of the negative log likelihood for logistic regression with gradient descent. Use the gradient of the sigmoid, p(1−p), times the input gradient with respect to beta, x.

Explore how beta parameters in logistic regression define log odds and odds ratios, with exponentials converting sums to a multiplicative probability ratio, and how dummy encoding shifts the intercept.

extend binary logistic regression to multi nominal logistic regression for three or more classes, using one-hot targets, class-specific intercepts, and softmax to produce class probabilities.

Extend linear classification to nonlinear problems by expanding the design matrix with polynomial features, enabling more complex decision boundaries that separate classes.

Use a roc curve to evaluate a binary classifier by plotting true positive rate against false positive rate as the threshold varies, highlighting the area under the curve.

Use the confusion matrix to benchmark classifier performance by showing true positives, true negatives, false positives, and false negatives, and to compute accuracy, precision, recall, and f1 score.

Explore logistic regression as a linear method for binary classification, visualize a two-variable dataset, and build a binary logistic regression model to separate two classes.

Create a binary logistic regression model with optional intercept and gradient descent optimization, provide fit, predict_proba, and predict methods that use sigmoid on the logit.

Demonstrate logistic regression classification by standardizing the design matrix with a standard scaler, fitting the model, and visualizing decision boundaries and probability contours for x1 and x2.

Extend logistic regression to multi-class with one-hot targets and softmax, update a class-by-feature weight matrix via gradient descent, and predict by argmax.

Use a multi nominal linear regression model with one hot y, standardize the design matrix, and plot the resulting decision boundary with intercept and coefficients.

The lecture demonstrates using logistic regression to tackle a nonlinear classification task, showing why a linear boundary fails and how polynomial regression enables a nonlinear decision boundary.

Transform the design matrix with degree-two polynomial features and standard scaler, then fit logistic regression to produce a nonlinear decision boundary in binary classification.

Explore how L1 and L2 regularization in logistic regression reduces parameter magnitudes and simplifies the decision boundary, illustrated with training and test data and polynomial regression.

Create a six by six plot of the data to show how adjusting the probability threshold changes the logistic regression decision boundary and model sensitivity.

Load the 60,000 28 by 28 digit images, convert to a 60,000 by 784 design matrix, and train a logistic regression model on 3,000 samples to classify digits.

Train a logistic regression model with a pipeline and standard scaler, optimize lambda via cross-validation with l2 regularization, and evaluate with accuracy, precision, recall, f1, and a confusion matrix.

Visualize the ten class coefficients of a logistic regression classifier as 28x28 images, showing which 784 pixels drive each digit decision from 0 to 9.