Deep Learning Bootcamp: Neural Networks with Python, PyTorch

Learn deep learning from basics to practical implementation with Python, PyTorch, and TensorFlow, featuring theory, hands-on coding, quizzes, activities, and projects like iris dataset and brain MRI CNN.

Meet Zia, the lead instructor at AI Sciences, guiding beginners through deep learning implementations with PyTorch in the deep learning bootcamp.

This lecture frames a hiring decision using deep neural networks, with two features, test scores and academic marks, to compare candidates and decide whom to hire.

Explore how increasing data improves decision reliability in a simple data-driven hiring example, distinguishing hired (green) from not hired (red) candidates using academia and test scores, and handling boundary cases.

Learn how neural networks define a decision boundary to classify employees, drawing a line or shape that determines who gets hired.

Explore how a linear equation with a bias creates a decision boundary to classify data, using x and y axes and a zero threshold.

Demonstrate the vectorized linear equation for binary decisions with w, x, and bias b producing y hat = w x + b. Apply a zero threshold to accept or reject, placing points on or above the line.

Explore how adding features beyond two dimensions leads to a 3D feature space and a hyperplane classifier using weights and bias to separate classes.

Explore how data resides in n-dimensional space with multiple features, weights, and a bias; multiply features by weights, sum them, and apply a greater-than-or-equal-to-zero rule to decide accept or reject.

Learn the perceptron model with features x1 and x2 (and any n features), weights w1...wn, and bias, to output yes or no via a zero threshold in Python.

Implement a basic perceptron in Python using NumPy matmul to compute x times w plus bias, and return binary value with a zero threshold; scale to many features with vectorization.

Explore how perceptrons implement logical gates by combining two independent perceptrons to realize and, or, and xor operations, and extend networks with additional units.

Train a perceptron by moving a decision line to separate red and green points, improving from a random line toward an ideal line.

explore perceptron training by adjusting a line defined by 2x1+3x2-7=0 to correct a misclassified red point, updating weights and bias gradually to avoid drastic shifts.

Examine how the learning rate, a hyperparameter between 0 and 1, scales weight updates using 0.1, shaping line changes and the correctly classified or misclassified points.

Learn how learning rate affects perceptron updates, with small rates slowing learning and near one speeds updates, and how labeling a positive point negative requires adding to the line equation.

Implement the perceptron algorithm in Python by looping over data with random weights and updating misclassified points using the learning rate alpha, adding for negative and subtracting for positive predictions.

Explore the basics of deep learning by reading a dataset, preparing features and labels, and visualizing data with a scatter plot to prep for the perceptron step.

Learn to implement the perceptron step function that updates weights and bias when points are misclassified, using the features x, labels y, and a learning rate.

Learn to train a perceptron in python with numpy, initializing random weights and bias, iterating over epochs with a learning rate, updating w and b, and plotting decision boundaries.

Visualize perceptron training by plotting lines from each result (with slope and intercept) on a scatter plot, exploring epochs and learning rate to find the best solution.

Learn why a single linear solution struggles with multi-criteria data, illustrated by hiring decisions, and why training data must reflect joint academic and test performance to avoid misclassification.

Demonstrates that linear solutions often fail, and that a non-linear curve provides a better fit to separate green and red points, illustrating a move from linear to non-linear solutions.

Understand error functions as the distance to a goal, where moving toward the goal reduces error until it becomes zero, illustrating how going down lowers error in learning.

Understand why discrete error functions offer limited guidance for improving a fitted line, and how continuous error functions enable meaningful improvements in line adjustments and height-based scenarios.

See how sigmoid replaces the step activation to yield continuous outputs between 0 and 1, using the formula 1/(1+exp(-z)) with w x + b.

Explore multi-class classification beyond binary tasks, moving from rain probabilities to distinguishing car, bicycle, or motorbike, and explain why a single line cannot separate three classes.

Explore how scores from a neural network, computed as w x + b, map to probabilities and why negative scores require an exponential transformation to positives.

Explain how softmax uses exponentials to form a probability distribution for multi-class classification. Implement a Python function that converts a list of numbers to softmax scores, like 0.67, 0.24, 0.09.

Define a softmax function over a list using NumPy, convert values to exponentials, divide by the sum of exponentials, and return a probability vector.

Describe one hot encoding to convert categorical classes into a binary matrix, creating a column per class (car, bike, bicycle) so models output a proper probability distribution.

Use maximum likelihood to compare models A and B by their probability predictions across decision boundaries and compute accuracy via the product of correct-label probabilities.

Explore the inverse relationship between error and probability, aiming to minimize error and maximize probability for accuracy, and learn that logarithms replace products with sums.

Explore how logarithms underpin cross entropy, using natural log to convert probabilities into an error measure that ranks models by likelihood, as a building block for deep neural networks.

Explore cross entropy as an error measure for two-class predictions, using probabilities, labels, and the negative log; learn the two-point formula and batch averaging in Python.

Explore the multi-class cross entropy formula with one-hot labels and class probabilities, and see how the extra summation generalizes binary cross entropy and ties to probability.

Explore cross entropy as an error measure inversely proportional to probability and implement a Python function with numpy that computes the cross-entropy for a given label y and model output.

Implement the sigmoid function using x as the feature input, returning 1 divided by 1 plus e to the minus x, implemented with numpy.

Define an output function using sigmoid of w dot x plus b, computed with numpy dot, to produce y hat from features X, weights w, and bias b, logistic regression.

Explore how the cross entropy error function is minimized by gradient descent to maximize the probability produced by sigmoid(w x + b) and reach the global minimum.

Explore convex functions, their ball-like curves with a single global minimum, and how derivatives reveal the direction to move for correct optimization.

Explore derivatives as the slope of a function with simple examples like f(a)=2a to illustrate positive slopes and height-to-width interpretation. See how derivatives drive gradient descent for convex functions.

Explore how derivatives guide gradient descent to minimize error, using slope with learning rate to update weights and move toward the goal on the x-axis.

Update weights and bias with gradient descent using cross-entropy. Logistic regression is a building block for deep neural networks.

Initialize random weights and a bias for logistic regression, then update them for each data point until the error nears zero, highlighting the small differences from the perceptron.

Demonstrate data visualization and a live logistic regression workflow, prepare a two-feature dataset with binary labels, plot admitted vs rejected, and outline weight updates using sigmoid, cross-entropy, and gradient descent.

Update weights and bias with gradient descent in Python, using the logistic regression output, derive the derivative of error from y and y hat, and adjust via the learning rate.

Implement logistic regression training in Python using gradient descent, with features and targets, initialize weights, compute loss, update weights, track accuracy, and visualize training progress.

Visualizes solution boundary, data points, and error over epochs to show gradient descent improving logistic regression, while comparing perceptron and logistic regression and exploring learning rate as a hyperparameter.

Compare perceptron and gradient descent, noting how misclassified points update weights and bias versus using every point to shape the decision boundary.

Learn why linear boundaries fail and how to form non-linear boundaries by combining multiple linear boundaries using addition, multiplication, and gates such as and, or, xor.

Combine two models by adding their green probabilities and apply the sigmoid function to convert the result into a valid probability that a point belongs to green.

Apply weighted sums to combine model predictions, assign higher weights to stronger models, incorporate a bias, and use sigmoid to estimate final class probabilities.

Transform linear model diagrams into neural network architectures by assigning weights and biases to perceptron units, merging submodels into a nonlinear network, and applying sigmoid activations.

Explore input, hidden, and output layers and how two or more hidden layers create a deep neural network, with variable neurons and hyperparameters shaping nonlinearity.

Explore binary versus multi-class classification, implementing an output layer with one neuron per class, using softmax to convert scores into probabilities and select the class with the highest probability.

Explore feed forward and back propagation in neural networks, combining linear models with weights, biases, and sigmoid activations. Understand layer notation and why this is not yet a deep network.

Describe feedforward in a deep neural network with three hidden layers, using weight matrices W1 to W4 and sigmoid activations to compute y hat, then minimize error.

Describe the core steps of deep learning: perform feed forward to produce y hat, compare with y, compute error, then apply back propagation to update weights toward a better model.

Explore back propagation in a network: use feed forward to compute y hat, measure error against y, and update weights with gradient descent using sigmoid of w x plus b.

Learn how to update a specific weight in a deep neural network using gradient descent, the error function, its partial derivative, and the learning rate.

Learn how the chain rule enables backpropagation in neural networks, computing partial derivatives of error with respect to weights through forward pass values like y hat, h1, and h2.

Implement a neural network with feedforward and backpropagation, derive derivatives for a single layer, and demonstrate vectorized coding using sigmoid derivative (sigma prime) for weight updates.

Build a simple neural network in Python and PyTorch to analyze student data with GRE, GPA, and rank; visualize admissions, then scale features and one-hot encode rank.

Learn to one-hot encode rank using pandas get_dummies with a rank prefix, drop the original column, and scale GRE and GPA columns in preparation for neural network input.

Scale the data by dividing GPA and GRE columns by maxima, store in processed data, then split into training and testing sets and separate features and labels for model training.

Split the data into 90/10 training and testing sets using numpy.random.choice, then drop the admit column to create features and use admit as the target for both sets.

Derive the sigmoid prime as sigma(x) times one minus sigma(x), compute output error with y and y hat using input x, and prepare to implement the neural network.

Develop a simple neural network with six input features and six weights, performing forward pass with sigmoid, backpropagation, and weight updates across epochs using a learning rate, without bias initially.

Test a neural network using updated weights on test features, compute sigmoid outputs, and classify with a 0.5 threshold; report 62% accuracy and discuss the single-neuron limitation and optimizations.

Examine underfitting and overfitting in neural networks, compare two models on training and testing sets, and pursue a model that neither underfits nor overfits.

Use early stopping by monitoring training and testing errors and stopping at the elbow when the testing error begins to rise, as shown at ten epochs.

Evaluate how a linear line classifies two points using equations x1+x2 and 10x1+10x2, and explore the role of bias and the sigmoid activation.

Compare two linear equations with weights w1 and w2 using a sigmoid classifier to separate points; larger weights boost margin and accuracy but can cause overfitting, necessitating regularization.

Penalize large weights to prevent overfitting by using L1 or L2 regularization with a small lambda. L1 yields sparsity for feature selection, while L2 supports continuous sparsity and aids training.

Apply dropout to prevent a single node from dominating by randomly deactivating neurons during forward passes, using PyTorch or TensorFlow settings like 0.2 or 0.5.

Understand how a complex error function can create multiple local minima, causing gradient descent to converge to false optima instead of the global minimum.

Explore how random restart helps neural network optimization escape local minima by comparing errors across multiple starting points, aiming for deeper or global minima and better trained weights.

Unpack the vanishing gradient problem caused by sigmoid activations, where tiny derivatives in backpropagation yield minimal weight updates and slow convergence in gradient descent.

Explore activation functions discussed for vanishing gradients, including sigmoid, tanh, and ReLU. Learn that tanh maps inputs to -1 to 1, while ReLU zeros negatives and preserves positives.

Explore a multi-class iris species project by loading and shuffling data, using four features, applying one-hot encoding to labels, and performing train/test splits for model training.

Initialize weights for a four-layer neural network with four inputs, two hidden layers (five and eight neurons), and three outputs; implement forward propagation with sigmoid activation and biases.

Implement backpropagation for a multi-layer neural network with sigmoid activation and its derivative, using numpy. Train the model by updating weights and biases through forward and backward passes.

Build a neural network workflow with predict and accuracy utilities, one-hot encoding, and a training wrapper using validation and epochs to monitor performance and prevent overfitting.

Test and debug a neural network by configuring layers from input features to outputs, setting learning rate 0.15 and 100 epochs, and evaluating training, validation, and testing accuracy.