A deep understanding of deep learning (with Python intro)

Learn the instructor's code-sharing policy: share the course code with attribution on GitHub or blogs, include the instructor’s name and course name on each file, and contact for commercial use.

Learn how artificial neural networks transform inputs into outputs using simple math, weights, biases, and non-linear activations. See how backpropagation tunes these bricks into powerful deep learning models.

Learn how deep learning models learn through forward propagation of data, backward propagation of error, and gradient descent that adjust weights across multiple layers based on feedback.

Explore the philosophy and implications of deep learning as an empirical, data-driven approach to intelligence, its contrast with first-principles reasoning, and the universal approximation theorem's role in modeling complex phenomena.

Apply an experimental science approach to deep learning by running parametric experiments. Vary learning rate and epochs, measure model accuracy, and assess feasibility and generalizability.

Explain why artificial neural networks diverge from real neurons, highlighting dendritic processing, neuron diversity, and the misleading label of artificial neurons.

Assess whether to use the Python coding appendix and the math section to build the deep learning skills, rooted in linear algebra, probability, and calculus.

Explore the math prerequisites of deep learning and implement them in NumPy and PyTorch, with practical Python examples in linear algebra, statistics, and back-propagation.

Examine spectral theories in mathematics and how deep learning decomposes complex systems into simple parts, via Fourier transform, SVD, dot products, non-linearities, and the simple–complicated–complex lens.

Define scalar, vector, matrix, and tensor, show how images become matrices and tensors, and explain data types with NumPy arrays and PyTorch tensors.

Translate reality into numbers for deep learning by distinguishing continuous reality from categorical reality. Apply dummy-coding and one-hot encoding to represent categorical data as matrices, enabling true/false, pass/fail labels.

Introduce the transpose operation and show how it converts rows to columns for vectors and matrices, using NumPy and PyTorch with row and column orientations and double transposes.

Explore the dot product, the core operation in deep learning, defined for vectors or matrices of the same shape, and implemented with NumPy and PyTorch in Python.

Learn how matrix multiplication is a structured dot product, requires inner dimensions to match, yields an M by K product, and is demonstrated with NumPy and PyTorch.

Explain the softmax function and its role in deep learning, demonstrate implementing it in NumPy and PyTorch, and show how outputs form probabilities that sum to one.

Explore how the logarithm function supports machine learning and optimization by using the natural log to stabilize tiny probabilities and smooth deep learning optimization.

Explore Shannon entropy and cross-entropy in deep learning, linking information theory to model performance. Learn entropy formulas, binary cross-entropy, and practical Python implementations with NumPy and PyTorch.

Learn to find minimum and maximum values in vectors and matrices, and apply arg min and arg max with NumPy and PyTorch, including zero-based indexing.

Explore mean and variance as measures of center and dispersion, compare variance with standard deviation and mean absolute difference, and implement these concepts in NumPy for deep learning regularization.

Explore sampling variability in deep learning by comparing random samples to the population mean, illustrating how law of large numbers and random sampling improve model generalization and reduce overfitting.

Explore how seeding NumPy and PyTorch enables reproducible randomness, controlling initial weights and sharing code, while recognizing when different initializations prevent the local minimum.

Explore how to compare deep-learning model architectures and parameter settings with the t-test, testing the null and alternative hypothesis and interpreting t-values, p-values, and SciPy results.

Explore how derivatives guide deep learning training and the intuition of slope and gradient descent. Learn to compute derivatives of polynomials, ReLU, and sigmoid activation functions with Python.

Identify local minima and maxima by setting the derivative to zero and using left and right sign changes to distinguish them for gradient descent, including vanishing gradient in deep learning.

Learn the product rule and chain rule for derivatives of interacting functions, with practical PyTorch examples and Python implementations to support understanding of gradient descent in deep learning.

Explore how gradient descent uses derivatives and the learning rate to minimize the loss function in deep learning, adjusting weights to reduce errors and approach the minimum.

Examine local minima and saddle points in high-dimensional gradient descent, and learn why deep learning often avoids them, with solutions like retraining from different starts or increasing model dimensionality.

Implement gradient descent in one dimension in Python with numpy and matplotlib, and explore how learning rate and training epochs affect convergence to the 0.5 minimum.

Explore gradient descent on the function cos(2 pi x) + x^2 to locate minima, understand starting value effects, including zero starting points and vanishing gradients in deep learning.

Move gradient descent from one to two dimensions, compute partial derivatives and the gradient with SymPy, and demonstrate the algorithm, the difficulties of gradient descent for finding global minima.

Convert gradient descent to gradient ascent in Python code for 2d problems by flipping the gradient sign. See how these methods direct the trajectory toward local maxima.

Explore gradient descent and its metaparameters—starting value, learning rate, and epochs—through three parametric experiments to design, run, and interpret results toward global and local minima.

Explore fixed versus dynamic learning rates in gradient descent, comparing gradient-based and time-based decay, and learn how RMSprop and Adam implement adaptive optimization for faster convergence.

Explore vanishing gradients and exploding gradients, and learn why they hinder gradient descent in deep learning, with strategies like residual networks, batch normalization, and regularization to prevent them.

Google Colab's revision history lets you view and restore earlier notebook versions, compare changes, and undo modifications using the file revision history and its dual-panel view.

Explore how a perceptron forms a weighted sum of inputs (dot product) with a bias, then passes through a nonlinear activation to form an ANN unit.

Explore the geometric view of neural networks and the perceptron, visualizing two-input feature space, separating hyperplanes, and how gradient descent learns weights for binary and continuous predictions.

Explore forward propagation of a perceptron, including linear weighted sums, bias, and the activation function. See how weights determine the separating hyperplane and how backpropagation learns them.

Learn how deep learning measures prediction errors with continuous and binarized errors, and how mean-squared error and cross-entropy losses drive model learning and cost-based optimization.

Explore how a perceptron evolves into a deep network and how backprop uses gradient descent through activation functions to minimize the loss with derivatives and the chain rule.

Understand how simple regression relates to the perceptron, then implement a PyTorch regression model to predict a continuous y from x using gradient descent and mean squared error.

Vary the regression slope m from -2 to 2 in 21 steps and train a deep learning model; compare mean squared error loss and the correlation between y and yHat.

Build and train an ANN for binary classification using two-dimensional qwerty data in PyTorch. Learn why sigmoid output with binary cross-entropy loss improves stability and discuss model accuracy and generalization.

Explore how learning rate affects binary classification in a simple neural network by running parametric experiments with BCEWithLogitsLoss in PyTorch, measuring loss and accuracy and logging variability across 50 repeats.

Build a multi-layer artificial neural network with hidden layers to improve qwerties classification, using two input features, ReLU activations, a sigmoid output, and explore learning rates with PyTorch.

Discover how removing nonlinear activation functions from hidden layers lets a linear model solve the qwerties classification problem with near-perfect accuracy, illustrating when linear solutions beat deep learning.

multilayer linear models collapse into one layer unless a non-linear activation or pooling is inserted between layers; non-linearity is essential to preserve multiple layers in deep learning.

Master multiclass classification with a multi-output neural network on the iris dataset, using a 4–64–64–3 PyTorch architecture and softmax cross-entropy training.

Take on a code challenge to merge two notebooks and build a three-group qwerties classifier in Python, training a two-input neural net with four hidden units and softmax output.

Investigate how hidden units influence deep learning performance by varying units in a one hidden layer model from 1 to 128, using iris and flowers data with accuracy plots.

Explore depth versus breadth in deep learning, and learn how model architecture affects total trainable parameters by counting weights and biases in PyTorch.

Learn to define deep learning models with nn.Sequential or by building your own PyTorch class, using init and forward to map layers and operations like ReLU and sigmoid.

Explore depth versus breadth in deep learning by building flexible models with varying hidden layers and units. Learn that architecture, not parameters alone, drives performance; deeper isn't always better.

Convert nn.Sequential to a custom class in Python intro by rebuilding a two-feature input to 16 hidden to one output with relu, sigmoid, and BCE loss with logits.

Explore how interpretable deep learning models are, from simple unit equations to the mystery of why they work. Weigh engineering goals against science aims and consider ethics in high-stakes decisions.

Explore what overfitting and underfitting mean in deep learning, how they affect generalization, and practical strategies like cross-validation and regularization to manage model complexity and researcher degrees of freedom.

Explore cross-validation in deep learning by learning training, development, and test sets, and how the independence assumption prevents overfitting while guiding model evaluation.

Explore generalization and its boundaries, identifying populations like adult humans, children, and animals, and how broader boundaries affect accuracy and training, dev, test sets with cross-validation.

Learn to implement cross-validation in python with manual random 80-20 train-test splits using numpy, balance classes, and evaluate deep learning models on iris data.

Explore cross validation with scikit-learn, splitting data into training and test sets, comparing accuracies, and noting how different partitions influence overfitting.

Learn to use PyTorch DataLoader and TensorDataset to create train and test loaders, train with mini-batches, and see how batch size impacts training time and accuracy.

Partition data into training, devset, and test sets using numpy and scikit-learn, then validate three-way splits for objective evaluation and reduced overfitting.

Show cross-validation on a simple regression problem in Python by splitting data into train and test sets, training on the training subset, and measuring mean-squared error on the test set.

Explore regularization in deep learning, showing how it prevents memorizing training examples and shapes representations to improve generalization through dropout, L1/L2, and data augmentation.

Switch PyTorch models between train and eval modes with net.train() and net.eval(). Use torch.no_grad() to disable gradients during evaluation and understand when dropout and batch normalization apply.

Dropout regularization improves generalization by randomly dropping units during training, typically with p=0.5, while PyTorch scales weights up during training and uses no scaling at test time.

Introduce dropout regularization in practice, evaluate its effect on model accuracy across dropout rates, and use mean smoothing to interpret rugged accuracy plots.

Explore dropout regularization in PyTorch with the iris dataset, showing how higher dropout can reduce accuracy on simple models, and prepare for weight regularization topics.

Learn weight regularization with L1 and L2 (lasso and ridge), how they shrink or sparsify weights, and how to choose lambda via cross-validation for deep learning.

Explore implementing L2 regularization in PyTorch using SGD weight_decay, train on the Iris dataset, and analyze how varying L2 levels affects training and test accuracy.

Show how to manually implement L1 regularization in PyTorch by summing absolute weights and biases, averaging the penalty, and adding it to the loss while accessing model parameters.

Explore mini-batches in deep learning with PyTorch, distinguishing batch and mini-batch concepts, and understand how batching provides regularization, speeds training, and optimizes forward and back propagation.

Explore mini-batch creation in PyTorch with DataLoader, including drop_last and batch_size effects on training and testing. Observe how different batch sizes impact learning speed and accuracy on the iris dataset.

Emphasize equal batch sizes in deep learning, illustrated by the qwerties doughnuts, to show how tiny test batches distort accuracy and how to fix.

Explore how mini-batch size affects model performance and learning speed through a parametric experiment from 2 to 64 on iris data, measuring training and test accuracy and plotting results.

Compare model parameters and meta parameters in deep learning, and see how architecture, hidden layers, units per layer, and optimization choices shape model performance.

Explore the wine quality dataset from UCI, clean and normalize features with z-score, remove outliers, binarize quality, and prepare PyTorch tensors and loaders for deep learning.

Explore a code challenge that builds a 11-feature binary wine quality model in PyTorch and analyzes how mini-batch size affects learning and overfitting.

Normalize data before input to deep learning models to keep inputs and activations on a consistent scale, enabling stable training; learn z-score, min-max scaling, and batch normalization as regularization.

Explore how data normalization through z-scoring boosts deep learning performance, as shown by Python notebooks with and without normalization and its impact on training and test accuracy.

Extend data normalization by applying batch normalization to inputs into every layer, learn gamma and beta during training, and regulate covariate shifts to improve learning in deep networks.

Explore batch normalization in PyTorch by comparing models with and without BN on the wine quality data, using z-score normalization and highlighting training and test accuracy.

Explore how batch normalization between the input and hidden layers affects a small neural network classifying qwertys, revealing near-perfect accuracy and proving batch normalization can help even shallow models.

Explore non-linear activation functions like ReLU, sigmoid, and tanh, their advantages and limitations for hidden and output layers, and how they affect gradients in PyTorch.

Visualize non-linear activation functions in PyTorch, compare ReLU, sigmoid, tanh and variants like ReLU6, LeakyReLU, Hardshrink, and learn how torch and torch.nn implementations differ.

In this lecture, the instructor empirically compares ReLu, tanh, and Sigmoid on the wine quality dataset using a fixed architecture with two hidden layers, highlighting test vs training performance.

Compare three ReLU variants—standard, Leaky, and ReLU-6—using torch.nn in PyTorch, and see Leaky and ReLU-6 train faster and often outperform the classic ReLU on wine quality data.

Explore deep learning with python by predicting residual sugar from wine features in a code challenge. Master mean-squared error, training vs. test splits, and correlation analysis.

Explore how loss functions drive deep learning training, comparing mean squared error and cross entropy, and examine how output layers use sigmoid, softmax, and logsoftmax to produce probabilities.

Explore loss functions in PyTorch, including MSELoss, BCELoss, BCEWithLogitsLoss, and CrossEntropyLoss, and learn to implement a custom loss function using forward.

Build a three-category qwerty predictor from x and y coordinates using a PyTorch neural network; explore code variants, visualization, and per-category accuracy to sharpen deep learning intuition.

Discover how optimizers modify stochastic gradient descent to train deep learning models. Learn mini-batch stochastic gradient descent and momentum, with a preview of RMSprop and Adam.

Explore momentum in stochastic gradient descent by adjusting the beta parameter in a PyTorch SGD optimizer on a three-category dataset, noting faster convergence with 0.9–0.95 and accuracy loss if high.

Explore RMSprop and Adam, adaptive optimizers that modulate per-weight learning rates by the history of gradient magnitudes and momentum, improving gradient descent.

Compare three optimizers—stochastic gradient descent, RMSprop, and Adam—on a simple qwerty dataset, training with a fixed learning rate and visualizing losses and accuracies to qualitatively assess performance.

Explore how learning rates affect three optimizers—gradient descent, RMSprop, and Adam—using logarithmically spaced rates from 1e-4 to 1e-1; observe RMSprop and Adam being robust while gradient descent is rate-sensitive.

Experiment compares L2 regularization with the Adam optimizer, testing lambda from 0 to 0.1; results show little benefit, with a possible slight improvement near 0.02.

Implement learning rate decay in PyTorch with a StepLR scheduler that halves the learning rate at set intervals, and compare dynamic versus static learning rates with weight decay.

Look to published models for similar data, but integrate your own knowledge to avoid overfitting. Use informed random or targeted grid searches to navigate the high-dimensional metaparameter space.