Math Behind LLMs, Transformers and Modern Computer Vision

Name: Math Behind LLMs, Transformers and Modern Computer Vision
Rating: 4.4 (1192 reviews)

From Multi Head Attention and Embeddings to Transformers, Vision Transformers, Modern Image Segmentation + LLMs and SAM

Created byPatrik Szepesi

Last updated 6/2026

English

What you'll learn

Mathematics Behind Large Language Models
Modern Image Segmentation
Positional Encodings
Compare CNNs and Vision Transformers mathematically
Compute prompt self-attention and image–prompt cross-attention
Multi Head Attention
Query, Value and Key Matrix
Attention Masks
Masked Language Modeling
Dot Products and Vector Alignments
Nature of Sine and Cosine functions in Positional Encodings
How models like ChatGPT work under the hood
Bidirectional Models
Context aware word representations
Vision Transformers
Word Embeddings
How dot products work
Modern Computer Vision
Understand quadratic complexity in Vision Transformers
Matrix multiplication
Programatically Create tokens
Derive self-attention, multi-head attention, and cross-attention from scratch
Analyze the full Vision Transformer pipeline
Break down the mathematics of Meta’s Segment Anything Model (SAM)
Understand prompt encoders in modern segmentation models

Course content

7 sections • 87 lectures • 11h 34m total length

What we are going to Cover in Natural Language Processing2:50
What we are going to Cover in Computer Vision1:49
Explore the shift from large-language models to computer vision and dive into vision transformers, patch and position encodings, and the segment anything model's image encoder, prompt encoder, and mask decoder.
(Optional) Mathematics Behind Backpropagation Course2:58
Course Resources0:03

Introduction to Tokenization11:44
Tokenization in Depth11:23
Programatically Understanding Tokenizations8:59
Encode text with the DistilBert tokenizer from pretrained, adding CLS and SEP tokens, applying max length and padding, and inspect input IDs, attention mask, and token type IDs.
BERT vs. DistilBERT5:07
Embeddings in a Continuous Vector Space7:50
Learn how embeddings convert tokens into 768-dimensional vectors learned via backpropagation. See how Bert base and DistilBert base use pre-trained embedding layers to enable fine-tuning for specific tasks.

Introduction to Positional Encodings5:20
How Positional Encodings Work4:18
Explore how positional encodings provide sequence order in transformers, add to token embeddings, and differ from rnn inputs, using a concrete example with I am a robot and noting dimensions.
Understanding Even and Odd Indicies with Positional Encodings10:31
Why we Use Sine and Cosine Functions for Positional Encodings5:15
Understanding the Nature of Sine and Cosine Functions10:04
Visualizing Positional Encodings in Sine and Cosine Graphs9:34
Explore custom color positional encodings using sine and cosine waves, showing how each embedding dimension maps to positions and repeats, with Plotly demonstrations.
Solving the Equations to get the Positional Encodings18:47
Learn how to derive and apply positional encodings for transformers using sine for even indices and cosine for odd indices, then add them to embeddings to form transformer inputs.

Introduction to Attention Mechanisms3:02
Query, Key, and Value Matrix18:20
Explore the attention mechanism at the heart of transformers, learning how query, key, and value matrices produce context-aware word representations through self-attention and attention scores.
Getting started with our Step by Step Attention Calculation7:09
Calculating Key Vectors20:44
Query Matrix Introduction10:34
Explore how the query matrix and query vectors compute attention scores by dot-product with key vectors, guiding how much each word attends to others in a sentence.
Calculating Raw Attention Scores21:59
Understanding the Mathematics behind Dot products and Vector Alignment13:56
Visualising Raw Attention Scores in 2 Dimensions5:56
Visualize raw attention scores in two dimensions for the word river, showing high attention to river, bank, and flooded due to vector alignment, dot products, and self-attention.
Note on Normalization Before Softmax0:19
While this topic will not be covered in detail in this course, it's useful to know that many deep learning architectures—particularly those involving attention mechanisms, such as in Transformers—apply normalization before the softmax function.
This normalization is typically done to scale the attention scores produced by the dot product of query and key vectors. By controlling the magnitude of these scores before applying softmax, the model avoids overly sharp probability distributions and promotes more stable training.
Converting Raw Attention Scores to Probability Distributions with Softmax9:32
Understanding the Value Matrix and Value Vector9:24
Calculating the Final Context Aware Rich Representation for the word "river"10:59
Understanding the Output1:57
Explore how the output context vector aggregates token vectors via attention to capture the word’s context from the entire sequence for downstream tasks like classification and translation.
Understanding Multi Head Attention12:13
Multi Head Attention Example, and Subsequent layers10:13
Masked Language Modeling2:33

Vision Transformers vs Convolutional Neural Networks5:45
Explore when to use vision transformers versus convolutional neural networks, highlighting global self-attention, patch-based processing, and scenarios like large data sets, segmentation, and medical imaging.
Quadratic Operations10:19
Introduction to Vision Transformers and Joint Training with Embeddings11:17
Understanding Attention Mechanisms, Brief Summary5:29
Understanding the Full Vision Transformer Pipeline17:36

Introduction to Prompt Encoders for SAM5:06
SAM AutoPrompt Mode16:00
Learn how SAM auto-prompt mode uses pre-learned object finder queries to drive prompt self-attention and cross attention over Vision Transformer embeddings for automatic mask generation.
SAM Manual Click Mode8:05
Vision Transformer Embeddings Inside SAM5:08
Explore a simplified path from a 2x6 input image through a vision transformer, using patch embeddings with 2d positional encodings and a single head self-attention encoder, then review prompt self-attention.
Calculating Attention Score for Vision Transformers in SAM17:21
Understand how vision transformers compute attention using queries, keys, and values, scale by dimension, apply stabilized softmax, and combine with values to produce context-aware patch representations, aided by residual connections.
How SAM is Trained8:17
Calculating Prompt Self Attention for SAM4:17
Learn how decoder prompt self-attention in auto-prompt mode lets three prompts share information using query, key, and value vectors, producing updated embeddings for image cross attention.
Prompt Image Cross Attention7:44
Image to Prompt Cross Attention6:04
Apply image to prompt cross attention by letting image tokens query the updated prompts to refine themselves. Strengthen true object patches and suppress background through two-way attention.
Finishing SAM Example Part 19:14
Finishing SAM Example Part 26:42
Explore how the IOU token and tiny prediction head assess mask quality in SAM, using mean pooling over prompts to rank and select the best, deduplicating overlaps.
Finishing0:04

Introduction to Our Simple Neural Network6:52
Explore a very simple neural network with input x, hidden layer h, and weights w1 and w2, and learn how backpropagation and gradient descent adjust toward the true label.
Why We Use Computational Graphs6:18
Conducting the Forward Pass6:54
Roadmap to Understanding Backpropagation2:46
Derivatives Theory4:28
Numerical Example of Derivatives13:43
Understanding Partial Derivatives8:02
Understanding Gradients3:52
Understanding What Partial Derivatives Do (Example)10:18
Introduction to Backpropagation5:00
Understanding the Chain Rule (Optional)7:35
Gradient Derivation of the Mean Squared Error Loss Function7:35
Visualizing the Loss Function + Gradients11:43
Using the Chain rule to Calculate the Gradient of w219:02
Using the Chain Rule to Calculate the Gradient of w14:38
Visualizing Gradient Descent10:14
Introduction to Gradient Descent6:14
Understanding the Learning Rate (Alpha)8:19
Moving in the Opposite Direction of the Gradient5:35
Calculating Gradient Descent by Hand8:47
Coding our Simple Neural Network Part 14:34
Coding our Simple Neural Network Part 27:20
Coding our Simple Neural Network Part 36:49
Coding our Simple Neural Network Part 45:09
Coding our Simple Neural Network Part 55:33
Introduction to Our Advanced Neural Network5:35
Conducting the Forward Pass4:32
Getting Started with Backpropagation4:54
Getting the Derivative of the Sigmoid Activation Function (Optional)7:42
Implementing Backpropagation with the Chain Rule4:55
Understanding How w3 Affects the Final Loss6:10
Calculating Gradients For Z17:42
Understanding How w1 & w2 Affect the Loss5:06
Implementing Gradient Descent By Hand8:34
Coding our Advanced Neural Network Part (Implementing Forward Pass + Loss)7:05
Coding our Advanced Neural Network Part 2 (Implement Backpropagation)10:41
Coding our Advanced Neural Network Part 3 (Implement Gradient Descent)5:51
Coding our Advanced Neural Network Part 4 (Training our Neural Network)8:21

Requirements

Basic HS math(linear algebra)

Description

Welcome to Math Behind LLMs, Transformers and Modern Computer Vision, a rigorous deep dive into the mathematical foundations powering today’s most advanced AI systems.

This course is designed for learners who want more than intuition. We derive and analyze the core equations behind Large Language Models, Vision Transformers, and modern image segmentation systems.

You will begin with tokenization and embedding mathematics, understanding how raw text becomes high-dimensional vector representations through algorithms like WordPiece. From there, we mathematically unpack the heart of transformer architectures: query, key, and value matrices, attention score computation, scaling behavior, and multi-head attention.

We examine attention masks, contextual encoding, and positional encodings — including the sine and cosine formulations that preserve sequence structure. You’ll build strong geometric intuition around vectors, dot products, cosine similarity, and dense embeddings.

The course then expands beyond language.

You’ll compare Convolutional Neural Networks with Vision Transformers, analyze quadratic attention operations, and walk through the complete Vision Transformer pipeline from patch embeddings to final predictions.

In an advanced section, we dissect the mathematics behind Meta’s Segment Anything Model (SAM). You will explore prompt encoders, self-attention, cross-attention between prompts and images, attention score computation in segmentation models, and how these systems are trained at scale.

By the end of this course, you won’t just understand how transformers work — you will understand why they work at the equation level across language and vision.

If you aim to build deep technical mastery and develop the mathematical intuition required for cutting-edge AI research and engineering, this course will elevate your expertise.

Who this course is for:

For ambitious learners aiming to reach the upper echelon of the programming world: This content is designed for those who aspire to be within the top 1% of data scientists and machine learning engineers. It is particularly geared towards individuals who are keen to gain a deep understanding of transformers, the advanced technology behind large language models. This course will equip you with the foundational knowledge and technical skills required to excel in the development and implementation of cutting-edge AI applications

Math Behind LLMs, Transformers and Modern Computer Vision

What you'll learn

Explore related topics

Course content

Course Overview4 lectures • 8min

Tokenization and Multidimensional Word Embeddings5 lectures • 45min

Positional Encodings7 lectures • 1hr 4min

Attention Mechanism and Transformer Architecture16 lectures • 2hr 39min

Mathematics Behind Vision Transformers5 lectures • 50min

Mathematics Behind Meta's SAM(Segment Anything Model)12 lectures • 1hr 34min

(Optional) Math Behind Backpropagation-The algorithm behind all Machine Learning38 lectures • 4hr 34min

Requirements

Description

Who this course is for: