Deep Learning for NLP - Part 1

Explore deep learning for NLP basics, including one-hot encoding and distributional models, efficient softmax approximations, glove word embeddings, cross-lingual embeddings, and subword embeddings.

Explore one-hot encoding limitations and how singular value decomposition enables dense semantic embeddings through latent semantic analysis for word–topic relationships and topic modeling.

Learn word2vec, the early distributional embedding model using cbow and skip-gram to derive word embeddings from large unlabeled corpora, capturing semantic relationships and analogies.

Explore efficient softmax approximations for large vocabularies, including hierarchical, differentiated, and CNN softmax, to speed training and reduce parameters with tree-based or character-based embeddings.

Explore sampling-based approximations for softmax that speed up training with large vocabularies, using important sampling, target sampling, and noise-contrastive estimation to reduce the softmax computation.

Glove blends global co-occurrence statistics with local context windows to produce word embeddings from co-occurrence matrices, using a cross-entropy objective and negative sampling to outperform count-based models.

Explore cross-lingual word embedding models that map multilingual vocabularies into a shared embedding space, comparing monolingual mapping, pseudo cross-lingual, cross-lingual training, and joint optimization methods.

Explore sub-word level embeddings for deep learning in NLP, including byte-pair and wordpiece encodings, fasttext, and sentencepiece, to handle out-of-vocabulary words and morphology.

Trace word embeddings from simple one-word encoders to semantic models, covering skip-gram, negative sampling, hierarchical softmax, co-occurrence matrices, cross-lingual embeddings, cbow, and sentencepiece.

Explore traditional language models and next word prediction models, and examine recurrent neural networks and image captioning networks, along with memory cell based models like Lithium's and Giuse.

Explore traditional n-gram language models and nnlm to understand sequence probability. Examine applications in spellchecking, speech recognition, and translation, plus RAM and context limits.

Learn how recurrent neural networks share weights across time steps, update hidden states with current input, and train via backpropagation through time for language modeling tasks using entropy loss.

Examine how image captioning uses a cnn encoder and a decoder to generate captions from images, trained with supervised data and backpropagation, with domain-specific success in eyecare.

Explore bidirectional and stacked recurrent neural networks to capture information from both past and future contexts, addressing vanishing and exploding gradients to improve long-term dependency modeling in NLP.

Learn how long short-term memory networks use input, forget, and output gates to control memory and combat vanishing gradients, enabling deep, bidirectional architectures and wide NLP applications.

Explore gated recurrent units (GRUs), which use update and reset gates to manage memory and capture long-range dependencies between past state and current input.

Motivate why recurrent models are essential for sequence modeling, explore bidirectional networks and image captioning, and discuss memory cells and memory-controlled architectures, noting older memory approaches faded.