Languages Models in Natural Language Processing

Recurrent neural networks are a specific kind of artificial neural network created to work with sequential data. It is utilized particularly in activities involving natural language processing, such as language translation, speech recognition, sentiment analysis, natural language generation, and summary writing. Unlike feedforward neural networks, RNNs include a loop or cycle built into their architecture that acts as a “memory” to hold onto information over time. This distinguishes them from feedforward neural networks.

Advantages

When it comes to natural language processing (NLP) text categorization tasks, recurrent neural networks (RNNs) provide the following benefits:

• Sequential Information Processing

• Variable-length Input

• Feature Learning

Disadvantages

For text categorization tasks in Natural Language Processing (NLP), Recurrent Neural Networks (RNNs) have certain drawbacks despite their benefits:

• Vanishing Gradient Problem

• Arrangement Preservation are missing

Types of RNN

1. Many to Many (Language Translation)

2. Many to One (Movie Rating)

3. One to One (Image to Image captioning)

4. One to Many (Baby name Generation)

Long Short-Term Memory (LSTM) is an enhanced version of the Recurrent Neural Network (RNN) designed by Hochreiter & Schmidhuber. LSTMs can capture long-term dependencies in sequential data making them ideal for tasks like language translation, speech recognition and time series forecasting.

Unlike traditional RNNs which use a single hidden state passed through time LSTMs introduce a memory cell that holds information over extended periods addressing the challenge of learning long-term dependencies.

Problem with Long-Term Dependencies in RNN

Recurrent Neural Networks (RNNs) are designed to handle sequential data by maintaining a hidden state that captures information from previous time steps. However they often face challenges in learning long-term dependencies where information from distant time steps becomes crucial for making accurate predictions for current state. This problem is known as the vanishing gradient or exploding gradient problem.

• Vanishing Gradient

• Exploding Gradient

• Lack of Long term dependencies.

Both of these issues make it challenging for standard RNNs to effectively capture long-term dependencies in sequential data.

LSTM Architecture

LSTM architectures involves the memory cell which is controlled by three gates: the input gate, the forget gate and the output gate. These gates decide what information to add to, remove from and output from the memory cell.

• Input gate: Controls what information is added to the memory cell.

The equation for the input gate is:

it=σ(Wi⋅[ht−1,xt]+bi)

C^t=tanh(Wc⋅[ht−1,xt]+bc)

• Forget gate: Determines what information is removed from the memory cell.

The equation for the forget gate is:

ft=σ(Wf⋅[ht−1,xt]+bf) ft​=σ(Wf​⋅[ht−1​,xt​]+bf​)
where:

• W_f represents the weight matrix associated with the forget gate.

• [h_t-1, x_t] denotes the concatenation of the current input and the previous hidden state.

• b_f is the bias with the forget gate.

• σ is the sigmoid activation function.

• Output gate: Controls what information is output from the memory cell.

The equation for the output gate is:

ot=σ(Wo⋅[ht−1,xt]+bo)

This allows LSTM networks to selectively retain or discard information as it flows through the network which allows them to learn long-term dependencies. The network has a hidden state which is like its short-term memory. This memory is updated using the current input, the previous hidden state and the current state of the memory cell.

Drawbacks

certain drawbacks despite their benefits:

• Exploding Gradient

• Overfitting

• Computational Complexity

Applications of LSTM

Some of the famous applications of LSTM includes:

• Language Modeling

• Speech Recognition

• Time Series Forecasting

• Anomaly Detection

• Recommender Systems

• Video Analysis

An encoder-decoder is a neural network architecture commonly used in sequence-to-sequence (Seq2Seq) models, particularly in tasks involving natural language processing (NLP) and machine translation. It consists of two main components: an encoder and a decoder.

1. Encoder: The encoder takes an input sequence and processes it into a fixed-size representation called the “context vector” or “thought vector.” The input sequence can be a sentence, paragraph, or any sequential data. The encoder typically uses recurrent neural networks (RNNs) such as LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit) to capture the sequential dependencies of the input. It processes the input sequence step by step and summarizes the information in the context vector, which aims to capture the essential information of the input.

2. Decoder: The decoder takes the context vector produced by the encoder and generates an output sequence. It can be another sequence of different length, such as a translated sentence or a response in a chatbot. Like the encoder, the decoder often utilizes an RNN architecture. It takes the context vector as the initial hidden state and generates each element of the output sequence step by step. The decoder is conditioned on the context vector and previous generated outputs, allowing it to generate the output sequence based on the learned representation.

The encoder-decoder structure is key to transformer models. The encoder processes the input sequence into a vector, while the decoder converts this vector back into a sequence. Each encoder and decoder layer includes self-attention and feed-forward layers. In the decoder, an encoder-decoder attention layer is added to focus on relevant parts of the input.

1. For example, a French sentence “Je suis étudiant” is translated into “I am a student” in English.

The encoder consists of multiple layers (typically 6 layers). Each layer has two main components:

1. • Self-Attention Mechanism – Helps the model understand word relationships.

   • Feed-Forward Neural Network – Further transforms the representation.

Applications of Transformers

Some of the applications of transformers are:

1. NLP Tasks

2. Speech Recognition

3. Computer Vision

4. Recommendation Systems

5. Text and Music Generation

Need of BERT:

Traditional language models process text sequentially, either from left to right or right to left. This method limits the model’s awareness to the immediate context preceding the target word. BERT uses a bi-directional approach considering both the left and right context of words in a sentence, instead of analyzing the text sequentially, BERT looks at all the words in a sentence simultaneously.

Example: “The bank is situated on the _______ of the river.”

In a unidirectional model, the understanding of the blank would heavily depend on the preceding words, and the model might struggle to discern whether “bank” refers to a financial institution or the side of the river.

BERT, being bidirectional, simultaneously considers both the left (“The bank is situated on the”) and right context (“of the river”), enabling a more nuanced understanding. It comprehends that the missing word is likely related to the geographical location of the bank, demonstrating the contextual richness that the bidirectional approach brings.

BERT is designed to generate a language model so, only the encoder mechanism is used. Sequence of tokens are fed to the Transformer encoder. These tokens are first embedded into vectors and then processed in the neural network. The output is a sequence of vectors, each corresponding to an input token, providing contextualized representations.

When training language models, defining a prediction goal is a challenge. Many models predict the next word in a sequence, which is a directional approach and may limit context learning.

BERT addresses this challenge with two innovative training strategies:

1. Masked Language Model (MLM)

2. Next Sentence Prediction (NSP)

1. Masked Language Model (MLM)

In BERT’s pre-training process, a portion of words in each input sequence is masked and the model is trained to predict the original values of these masked words based on the context provided by the surrounding words.

In simple terms,

1. Masking words: Before BERT learns from sentences, it hides some words (about 15%) and replaces them with a special symbol, like [MASK].

2. Guessing Hidden Words: BERT’s job is to figure out what these hidden words are by looking at the words around them. It’s like a game of guessing where some words are missing, and BERT tries to fill in the blanks.

3. How BERT learns:

   • BERT adds a special layer on top of its learning system to make these guesses. It then checks how close its guesses are to the actual hidden words.

   • It does this by converting its guesses into probabilities, saying, “I think this word is X, and I’m this much sure about it.”

4. Special Attention to Hidden Words

   • BERT’s main focus during training is on getting these hidden words right. It cares less about predicting the words that are not hidden.

   • This is because the real challenge is figuring out the missing parts, and this strategy helps BERT become really good at understanding the meaning and context of words.

In technical terms,

1. BERT adds a classification layer on top of the output from the encoder. This layer is crucial for predicting the masked words.

2. The output vectors from the classification layer are multiplied by the embedding matrix, transforming them into the vocabulary dimension. This step helps align the predicted representations with the vocabulary space.

3. The probability of each word in the vocabulary is calculated using the SoftMax activation function. This step generates a probability distribution over the entire vocabulary for each masked position.

4. The loss function used during training considers only the prediction of the masked values. The model is penalized for the deviation between its predictions and the actual values of the masked words.

5. The model converges slower than directional models. This is because, during training, BERT is only concerned with predicting the masked values, ignoring the prediction of the non-masked words. The increased context awareness achieved through this strategy compensates for the slower convergence.

2. Next Sentence Prediction (NSP)

BERT predicts if the second sentence is connected to the first. This is done by transforming the output of the [CLS] token into a 2×1 shaped vector using a classification layer, and then calculating the probability of whether the second sentence follows the first using SoftMax.

1. In the training process, BERT learns to understand the relationship between pairs of sentences, predicting if the second sentence follows the first in the original document.

2. 50% of the input pairs have the second sentence as the subsequent sentence in the original document, and the other 50% have a randomly chosen sentence.

3. To help the model distinguish between connected and disconnected sentence pairs. The input is processed before entering the model:

   • A [CLS] token is inserted at the beginning of the first sentence, and a [SEP] token is added at the end of each sentence.

   • A sentence embedding indicating Sentence A or Sentence B is added to each token.

   • A positional embedding indicates the position of each token in the sequence.

4. BERT predicts if the second sentence is connected to the first. This is done by transforming the output of the [CLS] token into a 2×1 shaped vector using a classification layer, and then calculating the probability of whether the second sentence follows the first using SoftMax.

During the training of BERT model, the Masked LM and Next Sentence Prediction are trained together. The model aims to minimize the combined loss function of the Masked LM and Next Sentence Prediction, leading to a robust language model with enhanced capabilities in understanding context within sentences and relationships between sentences.

Why to train Masked LM and Next Sentence Prediction together?

Masked LM helps BERT to understand the context within a sentence and Next Sentence Prediction helps BERT grasp the connection or relationship between pairs of sentences. Hence, training both the strategies together ensures that BERT learns a broad and comprehensive understanding of language, capturing both details within sentences and the flow between sentences.

The architecture of BERT is a multilayer bidirectional transformer encoder which is quite similar to the transformer model. A transformer architecture is an encoder-decoder network that uses self-attention on the encoder side and attention on the decoder side.

1. BERTBASE has 12 layers in the Encoder stack while BERTLARGE has 24 layers in the Encoder stack. These are more than the Transformer architecture described in the original paper (6 encoder layers).

2. BERT architectures (BASE and LARGE) also have larger feedforward networks (768 and 1024 hidden units respectively), and more attention heads (12 and 16 respectively) than the Transformer architecture suggested in the original paper. It contains 512 hidden units and 8 attention heads.

3. This model takes the CLS token as input first, then it is followed by a sequence of words as input. Here CLS is a classification token. It then passes the input to the above layers. Each layer applies self-attention and passes the result through a feedforward network after then it hands off to the next encoder. The model outputs a vector of hidden size (768 for BERT BASE). If we want to output a classifier from this model we can take the output corresponding to the CLS token.

4. The tokenizer.encode method adds the special [CLS] – classification and [SEP] – separator tokens at the beginning and end of the encoded sequence. In the token IDs section, token id: 101 refers to the start of the sentence and token id: 102 represents the end of the sentence.

Application of BERT

BERT is used for:

1. Text Representation

2. Named Entity Recognition (NER)

3. Text Classification

4. Question-Answering Systems

5. Machine Translation

6. Text Summarization