Python: Write Your Own Deep Learning Framework From Scratch

Set up a Windows 10/11 environment for building a deep learning framework by installing Python 3.8+, creating a virtual environment, installing NumPy, and configuring PyCharm with the correct interpreter.

Set up a Ubuntu-based development environment for building a deep learning framework by installing Python, creating a virtual environment, and installing NumPy; configure PyCharm with the virtual interpreter.

Set up a macOS development environment for building a deep learning framework by installing Python, creating a virtual environment, and installing NumPy.

Explore the basic neural network structure with input, hidden, and output layers, and show how weights and bias drive forward propagation and backpropagation, while gradient descent minimizes the loss.

Explore building a simple deep learning framework from scratch by implementing a base function class, a sine subclass, and using NumPy to perform forward computations.

Implement the variable class to manage values in the deep learning framework, initialize the value as an ndarray, and enforce type checks with explicit error handling.

Transform the function class to work with variable objects instead of ndarrays, retrieving xvalue from x, passing it to forward, and returning a new y object with the computed yvalue.

Implement a to_array helper to ensure forward outputs are ndarrays. Use numpy is_scalar to convert scalars to ndarrays, keeping all results consistent for deep learning computations.

Explore computation graphs, forward propagation, and backward propagation to understand how gradients guide deep learning model training.

Apply the chain rule on a computation graph to compute gradients. Multiply the derivative from the previous node by the current node's local derivative to obtain partial f partial x.

Add a grad attribute and backward method to build a simple framework, perform backpropagation through a sine function using cosine, and test gradient flow on a sine-sine example.

Automate backpropagation in the framework by linking producer relationships on variable and function nodes. This enables automatic backward passes from y grad to x grad through the computation graph.

Implement a gradient check function that uses central difference numerical differentiation to compare backpropagated gradients with numerical gradients, validating with allclose on a sine of sine example.

Optimize backpropagation by moving gradient initialization into the backward method, initializing grad when none and eliminating manual setup, so calling backward handles preparation automatically.

Implement variable arguments support for forward to handle multiple inputs by collecting xvalues and unpacking with *, and test with a two-input add function.

Extend backward propagation to handle multiple inputs and outputs, using gys, * unpacking, to_tuple, and zip to bind derivatives to inputs, while applying add and sin derivative rules.

Refine the gradient check function to support partial derivatives by using a lambda to fix non-target inputs and compute partials with respect to x0 or x1.

Demonstrates backpropagation when the same input is used twice in y = x0 + x0, showing gradient accumulation to avoid overwriting, and verifies the correct result with a test.

Implement a clear_grad method to reset each variable's grad to none. This prevents gradient carryover between repeated calculations and ensures correct backpropagation.

Fix the backward method by enforcing a level-based processing order in the computation graph, introducing level attributes and a function set to correctly accumulate gradients.

Clear the grad attributes of intermediate nodes during backpropagation to focus on input derivatives x and prevent unused grads from cluttering the backward pass.

Implement the addition operation by overloading the add method in the var class, converting inputs to ndarray objects and var objects, then test with sine of x plus x.

Extend the add method with radd to support right-side operands for var objects, and set a high array priority to ensure var operations override ndarray; test with 3 plus x0.

Implement the subtraction operator in a deep learning framework, mirroring addition, and update the forward and backward passes to yield gradients 1 and -1 while testing the change.

Implement the multiplication operation in the Python deep learning framework, performing forward computation and back propagation to compute x and y gradients from upstream derivatives.

Develop division in the backward pass by deriving x and y derivatives, update the forward division, and add true div and rtrue div in the var class, then test.

Apply the negate operation in backpropagation to derive dz/dx as minus 1, and implement y = -x with backward = -gy.

Implement the power operation in a simple deep learning framework by adding a pow class with forward and backward methods, computing gradients for x and testing x^n.

Explore tensors as multi-dimensional data structures, learn element-wise operations on arrays and matrices, and adapt backpropagation gradients to tensor shapes.

Implement the reshape method to alter tensor shapes without changing values using NumPy reshape in the forward pass, and reshape gradients to the input shape in backpropagation.

Rewrite the gradient check to support tensor inputs by computing per-element derivatives with an elementwise loop, using nditer and in-place updates, summing y0 and y1, and storing results in grads.

Study how broadcasting adapts shapes in add, subtract, multiply, and divide operations. Learn to backpropagate by summing gradients along the opposite broadcast directions using sum to restore original shapes.

We summarize the sum_to helper function that compresses input x to a target out shape using sum with keepdims, handling delta dim and delta axis, and validating final shape.

Implement the sum method for tensors, handling axis and keepdims, use numpy's sum for forward computation, and restore the gradient via sum backward shape during backpropagation.

Examine the helper function sum_backward_shape and how it restores gradient shapes for backpropagation. Learn how it handles axis, keepdims, and broadcasting to align gy with x's shape.

Implement a matmul class to perform matrix multiplication using NumPy's dot function, compute X and W gradients via backpropagation with transposed weights, and verify with a gradient check.

Implement the transpose method for matrices and tensors using numpy transpose, handling axis arguments, backpropagation, and gradient reshaping, with forward and backward passes and gradient checks.

Implement the exp method for tensors and perform backpropagation on the exp node, using dz/dx = z for gradient computation. Use numpy exp to compute y = e^x.

Explore neural networks by building the basic structure with input, hidden, and output layers. Discover how weights and bias drive z computations through matrix multiplication and a sigmoid activation.

Explore mean squared error as the loss function for training a neural network, and apply gradient descent to update weights and biases to minimize loss.

Build a simple neural network to learn y = x^2 from 1000 one-dimensional inputs, with a hidden layer, backpropagation, and gradient descent minimizing loss over 5,000 iterations.

Implement the module class as the base for layers, enabling forward flow from input x to output z, and manage its own parameters rather than manual w1, b1, w2, b2.

Implement the module class to control data flow through the network, using a forward method for calculations on var objects, and a tweaked call method to pass inputs into forward.

Implement a parameter class and integrate it into a module class, using a set to track parameters, __setattr__ overrides, and recursive, yield-based parameter traversal with clear grad support.

Define a linear layer class that inherits from the module class, initializes weights and bias, handles input size, and uses forward to perform matrix multiplication within a deep learning framework.

Construct the MyNet class by composing two Linear modules inside the Module framework, manage initialization and forward flow, and train via gradient descent with automatic input size handling.

Implement the optimizer class and its SGD subclass to update network parameters using stochastic gradient descent. Initialize, set learning rate, and call update param and step during training.

Reorganize the framework by placing core components—var class, function class, and forward/backward calculations—into core, move helpers into helper, place module class and linear class into layers, and SGD into optimizer.

Reorganize the nanotorch project by moving code into dedicated helper, core, layers, functions, and optimizers modules, updating imports while preserving the network's training behavior.

Add a dataset and a dataloader to the NanoTorch framework, and test them with the MNIST handwriting dataset of 28 by 28 grayscale images (60,000 training, 10,000 test).

Increase test data to 30,000 and use the dataset and data loader to train the network with minibatches, 100-item batches, shuffled indices, and epoch-based average loss.

Implement a base dataset class in NanoTorch, defining init, prepare data, get item, and length; subclass to provide specific data, enabling uniform data and label handling in training.

Implement a dataloader class for the NanoTorch framework to automate batch retrieval, shuffling, and iteration, making training cleaner with batch x and batch labels.

Download and unzip the MNIST dataset from the resources, then use the train images and train labels to train your network and begin implementing the MNIST dataset class.

Implement the MNIST dataset class to load training and test data from gzip files and map data and label paths, then reshape 28x28 grayscale images for use in the framework.

Learn how softmax converts neural outputs into category probabilities and how cross-entropy computes loss for multi-category problems, with examples from MNIST and output slicing.

Implement the log method in a custom deep learning framework, with forward y equals log(x) and backward gradient gy by computing dz/dx = 1/x.

Implement slicing in your deep learning framework, covering forward slicing, backpropagation into sliced positions, and a getitem method for the var object using numpy zeros.

Implement the clip class to truncate values to min and max limits, apply forward clipping with numpy, and perform backpropagation using a mask to zero gradients for clipped elements.

Implement the softmax and cross entropy loss in nanoTorch by defining the softmax function, clipping probabilities, computing the log, selecting true labels, and averaging the scalar loss for multiclass problems.

Introduce data preprocessing classes for MNIST: convert raw unsigned int images to float, flatten images, and apply min-max scaling. Integrate transforms into the dataset to prepare data for training MyNet.

Import softmax cross entropy, replace mean squared error, set the network output to 10 classes for mnist, and increase the hidden layer to 32.

Implement an accuracy function for the MNIST classifier by extracting final predictions from y_predict and comparing them to true labels. Compute the average with numpy mean to measure training accuracy.

Implement the ReLU activation in the core network, perform forward passes with numpy maximum, apply a mask in backpropagation, and replace sigmoid to boost accuracy at 0.9 after six epochs.

Evaluate your neural network on the mnist test set using a test dataloader, compute and print the accuracy, and compare training versus test performance in your from-scratch deep learning framework.

Implement the MLP class as a building block of deep learning, featuring input, hidden, and output layers. Refactor the existing MyNet into the MLP to underpin the framework.

Implement the MLP class in a models file, copy MyNet, import linear and ReLU, build L1, L2, L3 with h1, h2, and out, and train with sizes 64, 32, 10.

Explore and build your own convolutional neural network (cnn), the most widely used architecture for image tasks, and test it on the mnist dataset to observe its performance.

Explore the convolutional neural network from input to output, covering conv operations, padding, stride, and multi-channel feature maps to understand output shapes.

Implement a cnn class in NanoTouch with two conv layers (3x3 kernels, stride 2, padding 1) and linear layers. Reshape 3d conv outputs to 2d before feeding into layers.

Implement guideconfOutsize and Pair helper functions in the helper file to compute the CNN output size from input, kernel, stride, and padding using integer division for NanoTorch.

Implement the conf2d layer in the cnn class, initializing parameters, weights, and bias. Forward passes use stride and padding, with the conf2d function handling core convolution and backpropagation.

Explain the theory behind conv2d, showing how to reshape input and kernel into matrices and compute results by matrix multiplication for multi-channel convolutions, producing feature maps.

Implement the conv2d function in nanoTorch, preparing inputs and weights, applying stride and padding, reshaping kernels, and performing matrix multiplication to produce output maps with shape n, oc, oh, ow.

Implement Im2Mat class in the core file for a CNN in NanoTorch. Use ImageToMatrix in the forward method and MatrixToImage in the backward method, with kernel size, stride, and padding.

Master image to matrix and matrix to image in a CNN class with im2col, including batch size, padding, and stride, to enable efficient convolution via matrix multiplication.

Import CNN components such as Conf2D, ImageToMatrix, and MatrixToImage, apply Flatten preprocessing, and evaluate on the Amnesty data site using the CNN instead of the MLP, achieving rising accuracy.