A deep understanding of deep learning (with Python intro)

Learn the instructor's code-sharing policy: share the course code with attribution on GitHub or blogs, include the instructor’s name and course name on each file, and contact for commercial use.

Learn how artificial neural networks transform inputs into outputs using simple math, weights, biases, and non-linear activations. See how backpropagation tunes these bricks into powerful deep learning models.

Learn how deep learning models learn through forward propagation of data, backward propagation of error, and gradient descent that adjust weights across multiple layers based on feedback.

Apply an experimental science approach to deep learning by running parametric experiments. Vary learning rate and epochs, measure model accuracy, and assess feasibility and generalizability.

Explore the math prerequisites of deep learning and implement them in NumPy and PyTorch, with practical Python examples in linear algebra, statistics, and back-propagation.

Introduce the transpose operation and show how it converts rows to columns for vectors and matrices, using NumPy and PyTorch with row and column orientations and double transposes.

Explore the dot product, the core operation in deep learning, defined for vectors or matrices of the same shape, and implemented with NumPy and PyTorch in Python.

Learn how matrix multiplication is a structured dot product, requires inner dimensions to match, yields an M by K product, and is demonstrated with NumPy and PyTorch.

Explain the softmax function and its role in deep learning, demonstrate implementing it in NumPy and PyTorch, and show how outputs form probabilities that sum to one.

Explore how the logarithm function supports machine learning and optimization by using the natural log to stabilize tiny probabilities and smooth deep learning optimization.

Explore Shannon entropy and cross-entropy in deep learning, linking information theory to model performance. Learn entropy formulas, binary cross-entropy, and practical Python implementations with NumPy and PyTorch.

Learn to find minimum and maximum values in vectors and matrices, and apply arg min and arg max with NumPy and PyTorch, including zero-based indexing.

Explore sampling variability in deep learning by comparing random samples to the population mean, illustrating how law of large numbers and random sampling improve model generalization and reduce overfitting.

Explore how seeding NumPy and PyTorch enables reproducible randomness, controlling initial weights and sharing code, while recognizing when different initializations prevent the local minimum.

Explore how to compare deep-learning model architectures and parameter settings with the t-test, testing the null and alternative hypothesis and interpreting t-values, p-values, and SciPy results.

Explore how derivatives guide deep learning training and the intuition of slope and gradient descent. Learn to compute derivatives of polynomials, ReLU, and sigmoid activation functions with Python.

Identify local minima and maxima by setting the derivative to zero and using left and right sign changes to distinguish them for gradient descent, including vanishing gradient in deep learning.

Learn the product rule and chain rule for derivatives of interacting functions, with practical PyTorch examples and Python implementations to support understanding of gradient descent in deep learning.

Explore how gradient descent uses derivatives and the learning rate to minimize the loss function in deep learning, adjusting weights to reduce errors and approach the minimum.

Implement gradient descent in one dimension in Python with numpy and matplotlib, and explore how learning rate and training epochs affect convergence to the 0.5 minimum.

Explore gradient descent on the function cos(2 pi x) + x^2 to locate minima, understand starting value effects, including zero starting points and vanishing gradients in deep learning.

Convert gradient descent to gradient ascent in Python code for 2d problems by flipping the gradient sign. See how these methods direct the trajectory toward local maxima.

Explore vanishing gradients and exploding gradients, and learn why they hinder gradient descent in deep learning, with strategies like residual networks, batch normalization, and regularization to prevent them.

Google Colab's revision history lets you view and restore earlier notebook versions, compare changes, and undo modifications using the file revision history and its dual-panel view.

Explore how a perceptron forms a weighted sum of inputs (dot product) with a bias, then passes through a nonlinear activation to form an ANN unit.

Explore the geometric view of neural networks and the perceptron, visualizing two-input feature space, separating hyperplanes, and how gradient descent learns weights for binary and continuous predictions.

Explore how a perceptron evolves into a deep network and how backprop uses gradient descent through activation functions to minimize the loss with derivatives and the chain rule.

Understand how simple regression relates to the perceptron, then implement a PyTorch regression model to predict a continuous y from x using gradient descent and mean squared error.

Vary the regression slope m from -2 to 2 in 21 steps and train a deep learning model; compare mean squared error loss and the correlation between y and yHat.

Build and train an ANN for binary classification using two-dimensional qwerty data in PyTorch. Learn why sigmoid output with binary cross-entropy loss improves stability and discuss model accuracy and generalization.

Build a multi-layer artificial neural network with hidden layers to improve qwerties classification, using two input features, ReLU activations, a sigmoid output, and explore learning rates with PyTorch.

Discover how removing nonlinear activation functions from hidden layers lets a linear model solve the qwerties classification problem with near-perfect accuracy, illustrating when linear solutions beat deep learning.

multilayer linear models collapse into one layer unless a non-linear activation or pooling is inserted between layers; non-linearity is essential to preserve multiple layers in deep learning.

Master multiclass classification with a multi-output neural network on the iris dataset, using a 4–64–64–3 PyTorch architecture and softmax cross-entropy training.

Take on a code challenge to merge two notebooks and build a three-group qwerties classifier in Python, training a two-input neural net with four hidden units and softmax output.

Explore depth versus breadth in deep learning by building flexible models with varying hidden layers and units. Learn that architecture, not parameters alone, drives performance; deeper isn't always better.

Convert nn.Sequential to a custom class in Python intro by rebuilding a two-feature input to 16 hidden to one output with relu, sigmoid, and BCE loss with logits.

Explore generalization and its boundaries, identifying populations like adult humans, children, and animals, and how broader boundaries affect accuracy and training, dev, test sets with cross-validation.

Partition data into training, devset, and test sets using numpy and scikit-learn, then validate three-way splits for objective evaluation and reduced overfitting.

Show cross-validation on a simple regression problem in Python by splitting data into train and test sets, training on the training subset, and measuring mean-squared error on the test set.

Explore regularization in deep learning, showing how it prevents memorizing training examples and shapes representations to improve generalization through dropout, L1/L2, and data augmentation.

Switch PyTorch models between train and eval modes with net.train() and net.eval(). Use torch.no_grad() to disable gradients during evaluation and understand when dropout and batch normalization apply.

Introduce dropout regularization in practice, evaluate its effect on model accuracy across dropout rates, and use mean smoothing to interpret rugged accuracy plots.

Learn weight regularization with L1 and L2 (lasso and ridge), how they shrink or sparsify weights, and how to choose lambda via cross-validation for deep learning.

Explore implementing L2 regularization in PyTorch using SGD weight_decay, train on the Iris dataset, and analyze how varying L2 levels affects training and test accuracy.

Show how to manually implement L1 regularization in PyTorch by summing absolute weights and biases, averaging the penalty, and adding it to the loss while accessing model parameters.

Explore mini-batches in deep learning with PyTorch, distinguishing batch and mini-batch concepts, and understand how batching provides regularization, speeds training, and optimizes forward and back propagation.

Explore how mini-batch size affects model performance and learning speed through a parametric experiment from 2 to 64 on iris data, measuring training and test accuracy and plotting results.

Compare model parameters and meta parameters in deep learning, and see how architecture, hidden layers, units per layer, and optimization choices shape model performance.

Normalize data before input to deep learning models to keep inputs and activations on a consistent scale, enabling stable training; learn z-score, min-max scaling, and batch normalization as regularization.

Explore how data normalization through z-scoring boosts deep learning performance, as shown by Python notebooks with and without normalization and its impact on training and test accuracy.

Extend data normalization by applying batch normalization to inputs into every layer, learn gamma and beta during training, and regulate covariate shifts to improve learning in deep networks.

Explore batch normalization in PyTorch by comparing models with and without BN on the wine quality data, using z-score normalization and highlighting training and test accuracy.

Explore how batch normalization between the input and hidden layers affects a small neural network classifying qwertys, revealing near-perfect accuracy and proving batch normalization can help even shallow models.

Visualize non-linear activation functions in PyTorch, compare ReLU, sigmoid, tanh and variants like ReLU6, LeakyReLU, Hardshrink, and learn how torch and torch.nn implementations differ.

In this lecture, the instructor empirically compares ReLu, tanh, and Sigmoid on the wine quality dataset using a fixed architecture with two hidden layers, highlighting test vs training performance.

Explore deep learning with python by predicting residual sugar from wine features in a code challenge. Master mean-squared error, training vs. test splits, and correlation analysis.

Explore how loss functions drive deep learning training, comparing mean squared error and cross entropy, and examine how output layers use sigmoid, softmax, and logsoftmax to produce probabilities.

Explore loss functions in PyTorch, including MSELoss, BCELoss, BCEWithLogitsLoss, and CrossEntropyLoss, and learn to implement a custom loss function using forward.

Build a three-category qwerty predictor from x and y coordinates using a PyTorch neural network; explore code variants, visualization, and per-category accuracy to sharpen deep learning intuition.

Explore RMSprop and Adam, adaptive optimizers that modulate per-weight learning rates by the history of gradient magnitudes and momentum, improving gradient descent.

Compare three optimizers—stochastic gradient descent, RMSprop, and Adam—on a simple qwerty dataset, training with a fixed learning rate and visualizing losses and accuracies to qualitatively assess performance.

Explore how learning rates affect three optimizers—gradient descent, RMSprop, and Adam—using logarithmically spaced rates from 1e-4 to 1e-1; observe RMSprop and Adam being robust while gradient descent is rate-sensitive.

Experiment compares L2 regularization with the Adam optimizer, testing lambda from 0 to 0.1; results show little benefit, with a possible slight improvement near 0.02.

Implement learning rate decay in PyTorch with a StepLR scheduler that halves the learning rate at set intervals, and compare dynamic versus static learning rates with weight decay.

Look to published models for similar data, but integrate your own knowledge to avoid overfitting. Use informed random or targeted grid searches to navigate the high-dimensional metaparameter space.