Machine Learning: Neural networks from scratch

Explore forward propagation through a neural network block, multiplying inputs by weights, adding bias, and applying an activation function like relu to build nonlinear representations; discuss universal approximation.

Explore implementing the forward pass of a multi-layer perceptron in Python using NumPy, including weight and bias initialization, matrix multiplication, and shaping inputs for forward propagation.

Implement a neural network forward pass using a ReLU activation, validate outputs with tests and torch comparisons, and structure activation functions in a dedicated module for a scalable MLP.

Construct a compound sequential neural network by chaining blocks into an MLP, implement forward passes, and save and load weights to accelerate predictions.

Learn to save and load neural network weights across building blocks using numpy, assign unique file paths for each block, and test cross-network loading to enable ready-made predictions.

Explore predicting MNIST digits by building a three-layer mlp (128–64–10) with relu activations, preprocess data by centering and scaling to -1..1, and load pretrained weights for classification.

Learn to classify MNIST digits by loading weights, reshaping 28x28 images to 784 vectors, predicting with argmax, and achieving about 97.7% accuracy prior to applying the softmax activation for probabilities.

Learn how the softmax activation converts neural network outputs into probabilities by exponentiating inputs and normalizing by their sum, preserving the argmax and accuracy in forward passes.

Explore gradient descent with a simple two-parameter model, compare forward passes and Jacobian derivatives, and see how weight updates minimize loss toward the optimal values.

Explore jacobian matrices in neural networks, deriving derivatives with respect to inputs, weights, and biases, and apply them to gradient descent for training.

Implement jacobians for an mlp during backward pass, including w.r.t input, biases, and weights; store forward activations, and handle activation derivatives and identity matrices.

Use the chain rule to relate the loss to each block's weights, propagating gradients through the forward pass via the Jacobians of the MLP blocks to compute dL/dW and dL/dB.

Modify backward function to propagate gradients from the output toward the network's beginning using the chain rule and jacobian terms, and compute delta w and delta b for gradient descent.

Define a mean squared error loss class with a forward method that computes the mean of (pred minus true) squared and a backward method for the gradient wrt predictions.

Explore implementing and testing mean squared error loss for regression and classification, compare numpy and PyTorch gradients, and verify backward propagation in a simple neural network to train weights.

Train a neural network from scratch using gradient descent: perform a forward pass, compute mse loss, backpropagate gradients, update weights, and reshape deltas to fit the weight matrices.

Create an optimizer class that updates weights with a learning rate and conducts step and backward passes across mlp blocks. Explore rates from 1e-5 to 1, noting convergence and divergence.

Train a neural network for diabetes regression using a 10-feature dataset, scale the target to -1..1, and optimize with stochastic gradient descent, exploring learning rates and batching.

Explore the log softmax activation for classification, derive its forward and jacobian, implement numerically stable log softmax using the log sum trick, and validate against PyTorch with tests.

Master the log-sum-exp trick to stabilize log softmax calculations by subtracting the maximum value and using centered sums, with scipy and numpy, for neural networks from scratch.

Apply negative log-likelihood loss to maximize the true label's predicted probability while minimizing loss via gradient descent, compute Jacobians, and use Jacobian-vector products for efficient, scalable image classification.

Batching an MLP: handle batch size inputs, compute per-batch jacobians, concatenate them, and average gradients to update weights toward MSE loss.

Modify the MSE loss to support batched inputs by averaging over elements and over the batch, using the batch size, and validate with tests at batch size five.

Explore batching in neural networks by implementing activation function tests, comparing mean versus sum in gradient computation, and validating logsoftmax and jacobians across batches.

Explore the Jacobian-vector product and memory-efficient backprop: store only the result, drop unnecessary Jacobian terms, and apply algebraic rewrites to speed up gradients for MNIST training.

Explore Xavier initialization for deep feedforward networks, comparing uniform and Gaussian initializations, and learn how proper initialization preserves gradient flow to enhance MNIST training.

Train a neural network from scratch on the Mnist dataset for image classification, using an optimizer with a 1e-3 learning rate and ReLU and log-softmax activations with batch training.

Share feedback and reviews to help improve this machine learning and neural networks course. Explore future topics like gan-generated digits and other machine learning courses that extend these concepts.