LLMs Foundations: Tokenization and Word Embeddings Models

Learn how tokens and context determine meaning, and examine the WordPiece tokenization algorithm, its training phase, vocabulary setup with special tokens, and splitting words into characters.

Learn how WordPiece tokenization builds meaningful two-character subunits by scoring pairs with a frequency-based co-occurrence measure, approximating statistical surprise to avoid splitting words into meaningless units.

Explore tokenization with the wordpiece tokenizer and Bert tokenizer in a hands-on session: tokenize text, inspect vocabulary size, tokens and token IDs, decode back, and note CLS and SEP tokens.

Explore how the continuous bag of words embedding uses vectors, matrices, and tensors to represent features, with CBOW's training framed as a matrix-based, simultaneous equation model.

Learn the foundation of cbow by predicting a target word from its context words, training embedding weights to capture word relationships in a three-layer neural network, initialized with one-hot encoding.

Understand how the cbow model trains by averaging one-hot context vectors from the wordpiece tokenizer, multiplying by weights, and using a linear, non-activated hidden layer to predict the target word.

Learn practical CBOW embedding: replace sparse one-hot inputs with token index lookups, average context embeddings to predict target words, and apply the w x + b formulation for training stability.

Map one-hot vectors to dense word embeddings via the embedding layer. Predict the target word from averaged context embeddings with softmax and bias.

Explore a simplified cbow visualization where context words (one-hot vectors) are averaged, mapped via embeddings, and predicted with softmax, illustrated through forward and backward passes.

Explore the cbow model architecture details, from tokenizing a corpus and constructing a vocabulary to predicting target words from averaged context vectors and updating embedding weights through backpropagation.

Lookup embedding vectors by indices from the embedding matrix, replacing one-hot vectors, then average the context embeddings and feed the result into the hidden layer for CBOW-style processing.

Visualize how the embeddings matrix represents words as vectors and how a one-hot input selects a single row, turning embedding layers into lookup tables.

Explore the skip-gram word embeddings model, its comparison with cbow, input-output design for a target word to context words, and the role of negative sampling.

Provide a grand overview of word embeddings models, showing how a word’s meaning arises from its relationships with related words, captured by neural network weights in cbow and skipgram.

Define a training data prep function using text, a tokenizer, and a custom vocabulary with a two-token context window to map IDs to words and build cbow and skip-gram pairs.

Build cbow and skip-gram models with PyTorch by defining embedding layers, averaging context embeddings, and mapping to a vocabulary-sized output via a linear layer, trained with mini-batch gradient descent.

Build a skip-gram model in pytorch, using embedding and linear layers to predict context words from a center word, with batch training via a pytorch dataset and dataloader.

Implement a PyTorch training function that trains a CBOW or skip-gram model with a data loader, using cross-entropy loss and Adam optimizer across five epochs, monitoring average batch loss.

Train word embedding models on real data by tokenizing text, building a vocabulary, and training cbow and skip-gram models in PyTorch with a dataset, dataloader, batching, and embedding evaluation.

Practice testing word embeddings and semantic relationships by coding a skip-gram model in PyTorch to predict context words from a center word, highlighting inference versus training.

Practice building a mini large language model with skip-gram and cbow word embeddings in PyTorch, including training, embedding dimensions, and lowercase vocabulary handling.

Use pretrained word2vec embeddings with gensim to load a Google News model, then test word relationships via cosine similarity and analogies like king minus man plus woman to yield queen.

Discover how llms and ai chatbots rely on transformers, as large language models are transformers pre-trained on vast datasets; this course guides design to development.