Natural Language Processing From First Principles

Language transmits and stores vital information across generations, enabling knowledge to spread through reading and experience. Semantics and syntax create context-dependent meanings, challenging computers and driving natural language processing.

Examine how computers represent words beyond WorldNet, noting limitations of dictionaries and one-hot encodings. Learn to encode word meaning with dense, reduced-dimension vectors that capture similarity between words.

Explore how word vectors convert language into dense, continuous representations that capture similarity via context and dot products, moving beyond one-hot encodings to distributional semantics.

Explore how word vectors encode meaning from context, linking dot products to word co-occurrence probabilities in a corpus via skip-gram and continuous bag of words models.

Build a data set class to parse the Stanford movie reviews, skipping the header and line numbers. Normalize to lowercase, and generate vocabulary, word mappings, and word counts.

Learn how to augment a text dataset for word vector learning by applying frequency-based sub-sampling, dropping frequent words while preserving rare ones, and implementing a data augmentation function.

Select an augmented sentence at random, pick a context length from one to five, and extract the center word with its context.

Study how word vectors in high dimensions depend on theta to determine word probabilities. Use center and context vectors within a two-word window to compute sequence likelihood as a product.

Learn how to optimize word vectors by minimizing a cost function, using softmax and sigmoid probabilities to predict word occurrences in a corpus.

Explore how calculus guides optimization by using derivatives and gradients to minimize a cost function over the parameter space of word vectors, while navigating local minima.

Apply the chain rule to complex derivatives, use quotient and product rules, and handle logarithms, exponentials, and softmax, with a hands-on word vectors implementation for NLP.

Derive word2vec gradients using cross entropy loss with a softmax distribution, computing derivatives for center and outside word vectors, and applying the outer product for gradient updates.

Implement the skip gram algorithm by computing loss and gradients with softmax loss, converting words to indices, and accumulating gradients across context words.

Apply stochastic gradient descent to optimize word vectors using random mini batches of center words and contexts. Decay the learning rate to approach a global minimum.

Replace the naïve softmax loss with negative sampling to speed training by updating the center word and a few negative outside words drawn from a three-fourth power frequency distribution.

Explore the negative sampling loss function, derive its gradients for the center word, outside word, and negative samples, and implement efficient word vector training with testing compared to softmax.

Train word vectors with negative sampling and visualize results via the plot words function, optimizing with a one-million element table and focusing on central word vectors.

Use a paper's abstract, introduction, background, experiments, results, and conclusion to gauge significance, then map insights to code with data structures, algorithms, and unit tests.

Explore how the abstract and introduction describe a grand model learning syntax and semantics from unstructured text, using skip-gram with negative sampling and phrases to boost speed and quality.

Learn how the skipgram model learns word vectors by predicting context words from a center word, with a log probability objective, subsampling, negative sampling, and hierarchical softmax.

Compare hierarchical softmax, negative sampling, and sub sampling in analogical reasoning, and show how phrase skip-grams from 33 billion words reach about 72 percent accuracy.

Explore additive compositionality and comparisons in word vectors trained on 30 billion words, showing the skip phrase model outperforms others and produces meaningful long text representations.