Natural Language Processing: NLP With Transformers in Python

Explore the rise of neural networks, from deep learning to data-hungry models, and weigh adaptability and accuracy against brittleness and interpretability in natural language processing.

Explore recurrent neural networks and their recurrence for language, with time steps and output state preserving word order, then confront vanishing and exploding gradients in training.

Explore multi-head attention by applying parallel heads to generate diverse word representations, then concatenate and pass through a linear layer to produce attention-encoded tensors from position encoded embeddings.

Explain how positional encoding augments word embeddings to preserve token order in transformers, using sine and cosine waves and multi-head self-attention.

Explore how transformer heads customize bert for masked language modeling, classification, and question answering, using mask tokens, linear mappings to vocab or label spaces, softmax, and argmax.

Explore four attention mechanisms—encoder-decoder attention, self-attention, bidirectional attention, and multi-head attention—and learn how they work and where they’re used in transformer models.

Explore how the dot product computes alignment in encoder-decoder attention by comparing word vectors, visualized via heatmaps and 3d plots, with cosine similarity discussed as an alternative.

Explore dot-product attention with query, key, and value tensors from embeddings and linear layers for the encoder and decoder, using softmax to produce the final attention.

Explore multi-head attention in bidirectional self-attention within an encoder, using three heads, key/value/query tensors, and a linear projection to scaled dot-product attention.

Build your first transformer model with the flair library, loading the distilbert-based English sentiment model, tokenize text, process it, and extract sentiment labels with scores.

Learn to tokenize text for BERT using encode_plus, set a 512-token max length with truncation and padding, add special tokens, and produce input ids and attention masks.

Pass the input dictionary keys, including input ids, token type ids, and attention masks, to the model as keyword arguments, then apply softmax and use argmax to determine predicted sentiment.

Build an end-to-end NLP project using TensorFlow and transformers, covering data preprocessing with Kaggle data, tokenization, an input pipeline, and model training and predictions for sentiment or language classification.

Preprocesses Kaggle text data by reading and inspecting, optionally deduplicating, tokenizing with the BERT tokenizer to create input IDs and attention masks, and saving numpy arrays and one-hot labels.

Build a TensorFlow input pipeline by loading numpy arrays, creating a dataset with from_tensor_slices, and mapping to a dictionary of input IDs, attention mask, and labels for training.

Shuffle and batch the dataset with batch size 16, drop leftovers, split 90/10 into train and validation sets with take and skip, then save with tf.data.experimental.save.

Initialize a Bert base cased model with TF auto model, feed input ids and attention masks, add a classifier head for sentiment, train for three epochs, and save the model.

Discover how spaCy, the defacto standard for named entity recognition and part of speech tagging in Python, involves loading models and visualizing entities with the display module.

Pulls reddit posts via the Reader API, authenticates, requests up to 100 posts, and builds a pandas dataframe with subreddit, name, created utc, title, self text, and score.

Extract organization entities from reddit investing data using spaCy and pandas, build a function to process text and deduplicate results, and populate an organizations column for analysis.

Extract organization mentions from a dataframe, flatten nested lists, and count frequencies with collections.Counter to identify the top ten most frequently mentioned organizations.

Apply sentiment analysis to Reddit investing data with a flair distilbert model and spacy to extract organizations, then build a pandas dataframe of per-organization sentiment scores and rank them.

Master named entity recognition with RoBERTa in spaCy transformers, install spaCy transformers, load the transformer model, and compare RoBERTa-based ner to large spaCy models for accuracy.