Machine Learning: Neural networks from scratch

Explore implementing the forward pass of a multi-layer perceptron in Python using NumPy, including weight and bias initialization, matrix multiplication, and shaping inputs for forward propagation.

Implement a neural network forward pass using a ReLU activation, validate outputs with tests and torch comparisons, and structure activation functions in a dedicated module for a scalable MLP.

Explore predicting MNIST digits by building a three-layer mlp (128–64–10) with relu activations, preprocess data by centering and scaling to -1..1, and load pretrained weights for classification.

Learn how the softmax activation converts neural network outputs into probabilities by exponentiating inputs and normalizing by their sum, preserving the argmax and accuracy in forward passes.

Use the chain rule to relate the loss to each block's weights, propagating gradients through the forward pass via the Jacobians of the MLP blocks to compute dL/dW and dL/dB.

Modify backward function to propagate gradients from the output toward the network's beginning using the chain rule and jacobian terms, and compute delta w and delta b for gradient descent.

Explore implementing and testing mean squared error loss for regression and classification, compare numpy and PyTorch gradients, and verify backward propagation in a simple neural network to train weights.

Train a neural network from scratch using gradient descent: perform a forward pass, compute mse loss, backpropagate gradients, update weights, and reshape deltas to fit the weight matrices.

Train a neural network for diabetes regression using a 10-feature dataset, scale the target to -1..1, and optimize with stochastic gradient descent, exploring learning rates and batching.

Master the log-sum-exp trick to stabilize log softmax calculations by subtracting the maximum value and using centered sums, with scipy and numpy, for neural networks from scratch.

Modify the MSE loss to support batched inputs by averaging over elements and over the batch, using the batch size, and validate with tests at batch size five.

Explore batching in neural networks by implementing activation function tests, comparing mean versus sum in gradient computation, and validating logsoftmax and jacobians across batches.

Explore the Jacobian-vector product and memory-efficient backprop: store only the result, drop unnecessary Jacobian terms, and apply algebraic rewrites to speed up gradients for MNIST training.

Explore Xavier initialization for deep feedforward networks, comparing uniform and Gaussian initializations, and learn how proper initialization preserves gradient flow to enhance MNIST training.

Train a neural network from scratch on the Mnist dataset for image classification, using an optimizer with a 1e-3 learning rate and ReLU and log-softmax activations with batch training.

Share feedback and reviews to help improve this machine learning and neural networks course. Explore future topics like gan-generated digits and other machine learning courses that extend these concepts.