Udemy
    •  
    •  
    •  
    •  
    •  
    •  
    •  
    •  
Subscribe
Turn what you know into an opportunity and reach millions around the world.
Learn More
Your cart is empty.
Keep shopping
Deep Learning for NLP - Part 3
Rating: 4.7 out of 5(7 ratings)
116 students

Deep Learning for NLP - Part 3

Part 3: Sentence Embeddings, Generative Transformer Models
Created byManish Gupta
Last updated 7/2021
English
English [Auto],

What you'll learn

  • Deep Learning for Natural Language Processing
  • Sentence Embeddings: Bag of words, Doc2Vec, SkipThought, InferSent, DSSM, USE, MTDNN, SentenceBERT
  • Generative Transformer Models: UniLM, Transformer-XL and XLNet, MASS, BART, CTRL, T5, ProphetNet
  • DL for NLP

Course content

2 sections • 21 lectures • 3h 26m total length

  • Introduction1:51

    Trace sentence embeddings methods from recursive and CNN-based supervised models to unsupervised approaches like skip-thoughts, and explore multitask transformer-based and universal sentence embeddings.

  • Bag of Words approaches8:50

    Explore bag of words sentence representations and backoff methods, comparing word embeddings like glove and fastText, WMD, SRF, and power means for sentence embeddings and similarity.

  • Unsupervised methods: Doc2Vec9:41

    Explore unsupervised Doc2Vec methods for paragraph embeddings, including pv-dm and dbow, learning paragraph vectors with word vectors to predict next words within a paragraph.

  • Unsupervised methods: SkipThought and QuickThoughts9:44

    Explore unsupervised sentence embedding with skip-thought and quick-thought models, using encoder-decoder and discriminative approaches to predict surrounding sentences and evaluate semantic relatedness tasks.

  • Supervised method: RecNNs and Deep Averaging Networks9:05

    Explore recursive neural networks for tree-structured sentence embeddings and deep averaging networks for simple averaged word vectors, both trained with supervision for sentiment analysis, using dropout and softmax outputs.

  • Supervised method: InferSent10:23

    InferSent trains sentence embeddings via supervised natural language inference to produce transferable representations across SentEval benchmarks and tasks.

  • CNNs for semantic similarity: DSSM4:40

    Explore the deep semantic similarity model using cnn-based sentence embeddings and cosine similarity, with character 3-gram representations and convolutional layers for web search, translation, and qa.

  • Multi-Task Learning: USE11:15

    Explore multitask learning with Google's USC (universal sentence embedding), combining a transformer-based encoder and a deep averaging network to produce transferable sentence representations for classification, similarity, and conversational tasks.

  • Multi-Task Learning: MTDNN6:21

    MTDNN applies multitask learning through a shared deep neural network that processes sentence pairs, builds token embeddings, and trains task-specific heads across nine GLUE tasks, achieving strong fine-tuned results.

  • Multi-Task Learning: MILA/MSR Sentence Embeddings3:52

    Explore multi-task learning for MILA/MSR sentence embeddings with a recurrent encoder–decoder model. See competitive results using limited training data and semantically meaningful embeddings that separate subjective and objective classes.

  • SentenceBERT11:35

    Sentence-BERT develops semantically meaningful sentence representations for fast large-scale semantic similarity, clustering, and semantic search, outperforming simple mean or max pooling of word embeddings.

  • Summary2:39

    Provide an overview of sentence embedding methods, from distance based and word embeddings to unsupervised and supervised models, including recursive neural networks, deep average networks, and encoders.

Requirements

  • Basics of machine learning
  • Recurrent Models: RNNs, LSTMs, GRUs and variants
  • Basic understanding of Transformer based models and word embeddings

Description

This course is a part of "Deep Learning for NLP" Series. In this course, I will introduce concepts like Sentence embeddings and Generative Transformer Models. These concepts form the base for good understanding of advanced deep learning models for modern Natural Language Generation.

The course consists of two main sections as follows.

In the first section, I will talk about sentence embeddings. We will start with basic bag of words methods where sentence embedddings are obtained using an aggregation over word embeddings of constituent words. We will talk about averaged bag of words, word mover's distance, SIF and Power means method. Then we will discuss two unsupervised methods: Doc2Vec and SkipThought. Further, we will discuss about supervised sentence embedding methods like recursive neural networks, deep averaging networks and InferSent. CNNs can also be used for computing semantic similarity between two text strings; we will talk about DSSMs for the same. We will also discuss 3 multi-task learning methods including Universal Sentence Encodings and MT-DNN. Lastly, I will talk about SentenceBERT.

In the second section, I will talk about multiple Generative Transformer Models. We will start with UniLM. Then we will talk about segment recurrence and relative position embeddings in Transformer-XL. Then get to XLNets which use Transformer-XL along with permutation language modeling. Next we will understand span masking in MASS and also discuss various noising methods on BART. We will then discuss about controlled natural language generation using CTRL. We will discuss how T5 models every learning task as a text-to-text task. Finally, we will discuss how ProphetNet extends 2-stream attention modeling from XLNet to n-stream attention modeling, thereby enabling n-gram predictions.

Who this course is for:

  • Beginners in deep learning
  • Python developers interested in data science concepts
  • Masters or PhD students who wish to learn deep learning concepts quickly

Report abuse