Math Behind LLMs, Transformers and Modern Computer Vision

Explore how tokenization converts input text into tokens and token IDs, then creates word embeddings for Bert and Distilbert using wordpiece tokenization and a 30,000-token vocabulary.

Encode text with the DistilBert tokenizer from pretrained, adding CLS and SEP tokens, applying max length and padding, and inspect input IDs, attention mask, and token type IDs.

Compare bert and distilbert architectures, noting distilbert's six transformer layers, 66 million parameters, and faster inference with lower memory while retaining about 97% of bert's performance for most tasks.

Learn how embeddings convert tokens into 768-dimensional vectors learned via backpropagation. See how Bert base and DistilBert base use pre-trained embedding layers to enable fine-tuning for specific tasks.

Positional encodings insert word order into embeddings for Bert and Distilbert, enabling transformers to process entire sequences in parallel and clarifying the difference in their layer counts.

Explore how positional encodings provide sequence order in transformers, add to token embeddings, and differ from rnn inputs, using a concrete example with I am a robot and noting dimensions.

Learn how to compute four-dimensional positional encodings, applying sine to even indices and cosine to odd indices, using the I am a robot example.

Explore how sine and cosine positional encodings create a unique 784-dimensional fingerprint for word positions in transformers, enabling robust relative ordering in long sentences.

Explore custom color positional encodings using sine and cosine waves, showing how each embedding dimension maps to positions and repeats, with Plotly demonstrations.

Learn how to derive and apply positional encodings for transformers using sine for even indices and cosine for odd indices, then add them to embeddings to form transformer inputs.

Explore the attention mechanism at the heart of transformers, learning how query, key, and value matrices produce context-aware word representations through self-attention and attention scores.

Begin step by step attention calculation by transforming token embeddings with the query, key, and value matrices to illustrate how tokens like cls and bank interact.

Explore how the query matrix and query vectors compute attention scores by dot-product with key vectors, guiding how much each word attends to others in a sentence.

Calculate how the query matrix interacts with key vectors to produce raw attention scores, guiding context aware representations, then apply softmax and work with value vectors to finalize attention.

Visualize raw attention scores in two dimensions for the word river, showing high attention to river, bank, and flooded due to vector alignment, dot products, and self-attention.

Explore how the output context vector aggregates token vectors via attention to capture the word’s context from the entire sequence for downstream tasks like classification and translation.

Explore when to use vision transformers versus convolutional neural networks, highlighting global self-attention, patch-based processing, and scenarios like large data sets, segmentation, and medical imaging.

Learn why transformer attention scales quadratically with input length, as every token attends to all others, and how vision transformers and patches mitigate this growth in encoder and decoder contexts.

Explore how vision transformers split images into patches, project them to embeddings via a linear layer, and use a CLS token with learned positional encodings for image classification.

Explore how a vision transformer uses a CLS token as an aggregator of patch information through queries, keys, and values in multi-head attention, producing attention weights and an MLP head.

Learn how the segment anything model uses a vision transformer image encoder to create a detailed image embedding, and how the prompt encoder turns clicks into segmentation tokens.

Learn how SAM auto-prompt mode uses pre-learned object finder queries to drive prompt self-attention and cross attention over Vision Transformer embeddings for automatic mask generation.

Explore a simplified path from a 2x6 input image through a vision transformer, using patch embeddings with 2d positional encodings and a single head self-attention encoder, then review prompt self-attention.

Understand how vision transformers compute attention using queries, keys, and values, scale by dimension, apply stabilized softmax, and combine with values to produce context-aware patch representations, aided by residual connections.

SAM trains with a vision transformer image encoder, a prompt encoder, and a decoder using prompt self-attention and two-way attention, learned from the S1AB corpus with dice and cross-entropy losses.

Learn how decoder prompt self-attention in auto-prompt mode lets three prompts share information using query, key, and value vectors, producing updated embeddings for image cross attention.

Explore prompt image cross attention, where updated prompt embeddings query image tokens from the VIT encoder, producing attention scores that align prompts with image regions for geometric and semantic updates.

Apply image to prompt cross attention by letting image tokens query the updated prompts to refine themselves. Strengthen true object patches and suppress background through two-way attention.

This optional toy SAM demonstration walks through upsampling learned features to form patch-based mask maps, highlighting prompt-image cross attention and per-pixel mask logits.

Explore how the IOU token and tiny prediction head assess mask quality in SAM, using mean pooling over prompts to rank and select the best, deduplicating overlaps.

Explore a very simple neural network with input x, hidden layer h, and weights w1 and w2, and learn how backpropagation and gradient descent adjust toward the true label.

Learn how computation graphs break complex equations into modular steps to speed up backpropagation, reuse intermediate results, and enable parallel gradient computation with PyTorch or TensorFlow.

Learn how backpropagation uses partial derivatives and gradients with gradient descent to update neural network weights. We’ll build foundations from derivatives to partial derivatives and gradients, linking them to loss.

Visualize how the loss changes with predictions, using the gradient of the loss with respect to y hat to guide backpropagation and gradient descent.

Apply the chain rule to propagate gradients from z2 to the loss, using dY_hat/dZ2 = Y_hat(1−Y_hat) and confirming a negative gradient of -0.11222.

Implement a neural network with backpropagation and gradient descent, using Google Colab notebook, define a sigmoid and its derivative, perform forward and backward passes, and compute mean squared error loss.